Migrate and Refine Subject Import System from Third-Party 2026-07

Context

A prototyped Subject information import system was in development stage.
It was a script written in TypeScript, which transported data crawled from Third-Party's website to the backend every midnight.
Since the system orchestrating those importing jobs was not a backend, it was hard to trace why errors occurred.
I took on the task of restructuring this system to be more manageable.

Design

I analyzed the existing system and figured out an improved architecture.
I considered the locations of each feature according to the responsibility of each application.
I inherited the contract and logical processes as they were but refined and refactored them for better robustness.
I upgraded a one-time script to a Node.js server and assigned the crawling feature to it, while moving the other logic to the backend.
The reason it remained on an external Node.js server was that the library for web browsing is huge and the deployment cycle was independent of the backend.
I let the backend control and orchestrate the whole workflow. It requested crawled raw data from the Node.js server, then parsed it into refined data and finally committed it to the database.

Contributions

• Simplified the Node.js server's web browsing logic by switching the library from Selenium to Playwright.
• Improved the backend's validation logic for LLM's dynamic derived mapping response to preserve data invariants.
• Defined a dedicated thread for the work to avoid race conditions and release the Spring Scheduler's thread immediately.
• Restricted silent warnings that were easy to ignore and made them throw explicit errors.
• Reinforced its exception handling and improved traceability by adding logging.
• Built a pipeline to push the Node.js server image to a remote image registry (ECR) in AWS and configured permissions for managing the image.
• Focused on runtime-agnostic logic for easier testing.
• Conducted E2E tests in a local environment and staged them to the dev server.
• Documented and shared knowledge with colleagues.

Challenges

I reviewed the Spring Batch framework for this batch work but I didn't adopt it because:

• Many of the steps needed HTTP communications, such as web crawling. Since these communications took a long time, the transaction was hard to manage.
• It was well-suited for chunk-oriented processing but the size of data on this job was too small.