316 product cards into Word in one day
The client had an archive of 318 product folders from a UK online store of coffee equipment and was copying descriptions, specs and reviews into old Word files by hand. I built a parser that filled 316 .doc files following the client's sketch in about 2 hours of work — instead of more than 26 hours of manual copying.
What was broken
The client filled the catalogue by hand: open a product page, copy the text, paste it into Word. One card took about 5 minutes; across the whole archive that is more than 26 hours of monotonous work. He had done two files himself and stopped.
- A captcha at every step. The store asked to prove you’re human even when switching tabs on the same product page.
- Word broke the text. Text copied from the page was pasted as tables, so the client first moved everything to .txt and only then into Word.
- A rigid archive structure. Each product has its own numbered folder: a .txt with the URL, a .doc for the text, an Excel file and a photo folder. The txt, Excel and photos must not be touched.
- An old format. The result had to go into the same .doc (Word 97-2003), not .docx.
What each file had to contain
Following the client’s sketch of 7 screenshots: breadcrumbs, title, price, return guarantee, the Perfect for block, description, the list of drinks, specs from the Specifications tab (the visible part and the full table), instructions and reviews. Reviews: the oldest ones, from the last three pages of the site, without replies.
Constraint: 1 day for everything.
The solution
I split the work in two: data collection in the browser and processing on the computer. The site only sees a regular visitor session. Parsing and writing to Word happen locally, with no further requests to the site.
The site only sees a regular visitor session. Parsing and writing to Word happen locally, with no requests to the site.
Python goes through the archive twice: first it collects URLs for the browser, then it puts the collected data back into the same folders.
- Generator (Python, step 1). Walks the folders from a given number and reads the product URL from each .txt, skipping the category URL line. It builds the collect.js script from the URLs, and PowerShell puts it straight on the clipboard.
- Collector (JavaScript in the browser console). The console is a built-in browser tool for running a script on the open page. The collector fetches each product page and all its reviews from the store’s public API in turn. An API is an address the site itself uses to serve data to its pages.
- Polite to the site. A 1.5–3 second pause between products. On an error, 3 retries with a growing pause. Reviews come in batches of 100 with a pause between batches.
- Processor (Python, step 2). Takes the newest data.json from Downloads and parses the page with BeautifulSoup. It assembles the text in the order of the sketch and skips blocks the product page doesn’t have. Replies to reviews are dropped: they have no rating.
- Writing to Word. COM is the mechanism a program uses to drive an installed Word, like a user with a mouse. The text goes straight into N.doc and is saved in 97-2003 format. There are no tables, because the document gets clean text, not a copied page.
- Safety net. Without Word, LibreOffice takes over. If a file is open in Word, the script stops with a hint instead of damaging it.
- Report. report.csv: one row per folder with the title and counts of drinks, specs, reviews on the site and reviews taken. A card without a title or description gets a “check manually” status.
The whole run is two commands: generate the collector and process its result.
Development story
Explaining the captcha
What happened: the client didn’t understand my first explanation (“the script doesn’t click through tabs”) and asked to see it in action.
How we fixed it: that same evening I recorded a video: 3 products from three different categories, the product page and the finished .doc side by side.
Formatting
What happened: the client planned to fix the formatting by hand, doubting the parser could handle it.
How we fixed it: I matched the client’s sample: font sizes, bold headings, blank lines between blocks. Blocks missing on the product page don’t end up in the file.
Lessons
- A demo video convinces better than an explanation. The technical answer about the captcha didn’t work; 3 products on video settled the question completely.
- A per-item report pays off. In a minute, report.csv showed a card without a description and 11 duplicates the client didn’t know about.
- Don’t bypass protection — work like a regular visitor. The collector runs in a normal browser session with pauses, so the site has nothing to react to and no captcha appears.
Result
The client received 316 filled .doc files in the same folders, plus report.csv with a summary for each product. From receiving the archive to the finished result took about 2 hours — instead of more than 26 hours of manual copying. Blocks missing on the product page didn’t make it into the file, so there are no empty headings.
- The parser remains a ready tool: it’s tuned to the store’s structure and runs with two commands.
- Refreshing reviews or collecting other categories takes minutes, not a day.