Browsing the web is child’s play for us humans, but it becomes a real headache for AI agents… Because between dynamic content, unwanted popups and anti-bot systems, our artificial friends often find themselves helpless.
But the situation is about to change thanks to Steelan open-source API that promises to streamline the way AI agents interact with the web.
Why Steel will make your life easier?
Developing AI applications capable of interacting with the web poses many technical challenges. First there is dynamic content which requires the execution of JavaScript, then complex navigation paths involving several steps, but also the management of authentications and connections. And I’m not talking about the circumvention of anti-bot systems or the heavy infrastructure to maintain to manage headless browsers
Steel therefore arrives like the messiah in the soup ^^ to solve these problems with a completely redesigned approach, which will prevent you from tinkering with hazardous solutions, as you know how to do so well!
Steel’s superpowers in detail
Excellent session management
Steel automatically manages cookies and local storage, browser state between requests, but also connection persistence and automatic authentication processes.
Steel transforms web pages into easily digestible formats including clean and structured Markdown conversion, high fidelity screenshots, PDF generation and of course optimized extraction of relevant content which reduces up to 80 % the use of tokens.
Anti-detection features
- Automatic proxy rotation
- Fingerprint management
- Built-in stealth plugins
- Smart bypass of CAPTCHAs
Here are the different options for getting started with Steel:
Installation via Docker :
git clone https://github.com/steel-dev/steel-browser
cd steel-browser
docker compose up
Direct installation with Node.js :
npm run install
npm run dev
The different ways to use Steel
Quick mode with the Actions API
For simple reading tasks like extracting the content of a page:
curl -X POST http://0.0.0.0:3000/v1/scrape -H "Content-Type: application/json" -d '{ "url": "https://example.com", "waitFor": 1000 }' Advanced mode with custom sessions
For total control over your browsing:
curl -X POST http://0.0.0.0:3000/v1/sessions -H "Content-Type: application/json" -d '{ "options": { "proxy": "user:pass@host:port" } }' Compatibility with existing frameworks
Steel integrates perfectly with Puppeteer in order to have precise control of the browser but also Playwright for robust end-to-end testing or even Selenium for compatibility with existing scripts
Practical use cases
Here are some ideas of what you can do with Steel (and in style, of course!)
- Automated Shopping Assistant : Create an agent who compares prices, checks availability and makes purchases across different platforms.
- Smart content aggregator : Develop a system that collects and synthesizes information from multiple sources.
- Business Process Automation : Automate repetitive tasks like data entry or report generation.
Tips for getting the most out of Steel
Optimize your sessions
- Reuse sessions for similar tasks
- Clean up unused sessions regularly
- Use appropriate timeouts
Manage resources intelligently
- Limit the number of parallel sessions
- Configure allocated memory wisely
- Monitor CPU consumption
Maximize discretion
- Vary the navigation settings
- Use random delays
- Alternate proxies regularly
To go further
Steel offers advanced debugging tools like a live session viewer, session replay and even embeddings for analysis. In addition, the platform is constantly evolving, with an active community that contributes to its improvement so do not hesitate to contribute on Github or Discord.
So, the next time you need to develop an AI application that interacts with the web, at least you will know what to use to avoid the usual headaches.
Learn more about Steel Browser
