The situation

Some portals have no API. There is a web page, a login, a form, and a document behind it, and the only way to read that document is to be a browser. That happens more often than people expect, particularly with government, institutional and older enterprise systems.

The temptation is to scrape the HTML. The better approach is to drive a real browser - the same one a person would use - and treat the page as the interface it is.

The difference matters for reasons that are not obvious until you have tried both. A browser executes the JavaScript, so you get the rendered result, including content that only appears after hydration. A browser holds the session, so cookies, tokens and redirects behave exactly as they do for a person. And a browser can be watched - a person can watch the automation work and see it fail in the same way they would see it fail themselves.

The first rule: the target moves

The thing that breaks browser automation is not complexity. It is change. A portal gets a redesign, a button is renamed, a new step is added to a form, a class name changes because somebody tidied the CSS.

A script with a selector in it is a promise about a page that somebody else controls. The only question is when that promise breaks - and if the answer is "during a quarter-end", you will find out at the worst possible moment.

So the design principle is: every assumption about the page is in one place, and nothing else is allowed to look at the page.

The shape of the thing

A page description module. Every selector, every expected label, every expected field, in one file. Nothing else in the codebase contains a CSS selector or a button label. When the portal changes, one file changes.

A state check before every action. The script does not act on the assumption that the page is in a particular state - it checks, and it fails with a useful message when the state is not what it expected. "Expected the results table, found the login form" is an actionable error. "Element not found" is not.

A refusal to guess. If an element cannot be identified unambiguously, the script stops and says so. A script that falls back to "the first button on the page" will, one day, click something that submits a payment.

Isolation of the steps. Login, navigate, read, download, file. Each step takes what it needs and returns what it produces, and none of them reaches into another's internals. When the portal changes, the broken step is the only one that needs fixing.

Getting the data out

Two approaches, and they have different failure modes.

Structured extraction. Read the values out of the page - a table, a list, a set of fields. Fast, and it breaks when the markup changes. It is the right approach for data you need to use, because it goes straight into a database.

Downloading the document. When the target is a file - a PDF, a certificate, a statement - capturing the download is more faithful than parsing the page, and the file itself can be stored with a hash. The parsing of the file's contents is a separate step with its own tests.

The combination is usually right: extract the metadata from the page, download the document, and treat the file as the source of truth.

The parts nobody warns you about

Sessions expire. Long-running jobs hit a login wall in the middle. The script has to detect the session loss as a page state - a login form where a document was expected - and re-authenticate, rather than failing with a confusing null.

Downloads are not synchronous. A click that starts a download does not mean the file exists yet. The browser has to be told to wait for the file, and the failure mode of getting this wrong is a file that exists but is empty, which is worse than no file at all.

Rate and politeness. A portal that is a system of record for a real institution will be noticed if it is hit hard. A single session, a small delay between actions, and a clear log of what was done and when - both for the institution and for the person who has to debug it afterwards.

The login is the security boundary. Credentials for these portals are real, and they belong in the environment, not in the repository. The extraction layer should never be able to log them, and the error messages should never include them.

Being honest about what it produced

The last piece, and the one that matters most, is the record of what happened.

Each run writes down: when it ran, what it found, what it changed, what it skipped and why. A portal that has been scraped for a year with no log is a mystery, and when the portal changes there is no way to tell whether the data is stale, wrong, or missing for a reason nobody recorded.

The same honesty applies to the data itself. A field that the portal did not show is absent, not empty and never guessed. A document that could not be downloaded is recorded as failed, with the reason, and it is retried later - rather than quietly skipped.

The rules I would keep

• Keep every assumption about the page in one file. It is the only part that will need changing.
• Check the page's state before acting on it. Never assume you are where you think you are.
• Prefer ambiguity to a wrong guess. Stopping is a feature.
• Record what the run did, and what it could not do. A log is the difference between a tool and a rumour.
• Treat the credentials as the most sensitive thing in the system, and keep them out of the logs.

The implementation described in DocFlow Automator is organised exactly this way: the portal description is isolated, every step verifies the page's state, and the output is a record of what was done rather than a pile of files with no provenance.