Expected duration: 1 day or less **Please note that I will only respond to or accept someone who has answered the interview questions**
I have a RocketReach account and would like to automate the process of gathering contact information for a specific search query (e.g., "Ophthalmology"). My RocketReach account manager has confirmed that the website can be scraped as required, so I need a solution to automate this process effectively.
Requirements:
1. Phase 1: Automating "Get Contact Info"
The scraper should log in to my RocketReach account. Navigate to a specific search query (e.g., "Ophthalmology"). For every profile returned in the search results, the script should click the "Get Contact Info" button to unlock contact information for that profile. The script must handle large searches (e.g., 25,000 contacts spanning 500 pages). Pagination must be automated, ensuring that every profile on all pages is processed.
2. Phase 2: Scraping Contact Details
After Phase 1 is complete, I need the scraper (or a separate script) to return to the same search query. For every profile on all pages, the scraper should extract and compile the following details:
Full Name Job Title Company Work Email Personal Email Phone (if available)
Delay Handling: Contact information may take time to fully load after being unlocked. A delay should be programmed between scraping actions for each profile to ensure all data is fully available before moving to the next profile. The delay should mimic natural human interaction to avoid detection. The output should be saved to an Excel spreadsheet in an organized format with the above fields as columns. Additional Requirements: Error Handling: Ensure the scraper gracefully handles scenarios such as missing data, captchas, rate limits, or changes in website structure. Performance: Since the searches can be large, the script should be efficient and able to handle high volumes without crashing. Documentation: Provide clear instructions on how to run the script, along with any required dependencies or setup.
Deliverables:
Phase 1 Script: Automates the "Get Contact Info" action for all profiles in the search results. Phase 2 Script: Scrapes the unlocked contact details and exports them to an Excel spreadsheet, with delay handling implemented to ensure all data loads properly.
Documentation: Instructions on how to:
Install and run the scripts. Handle potential errors or limitations (e.g., captchas or rate limits). Modify the script for different search queries if needed.
Notes: Please ensure the script can handle long runtimes for large searches or include a mechanism to resume progress if interrupted. If you recommend a particular technology or approach (e.g., Selenium, BeautifulSoup, Puppeteer, etc.), please outline it in your proposal. I am open to suggestions on whether a single script or separate scripts for each phase would be better.
