Web scrapers and being stuck
I’ve recently been making a project due to my Upwork requests. The first project was a Facebook web scraper t thought it will be pretty easy. But I found out I had to have a facebook account and it when all downhill from there. The goal was to design an web scraper that takes the name of people from a facebook group then copy the name to search them on LinkedIn and save the URL if the person has a LinkedIn profile. So I imported my libraries and set up selenium to log in to Facebook after I made a whole need google account to I can use a new email address. After I when to work so the idea was to use selenium to grab the element and extract the text. But I hit a problem as i could not select using XPath or class. Which I had a similar issue with my Bloomberg project as I could not grab the element for the accept button. Because i was struggling to use selenium to get the job done i decided to switch to beautifulsoup but I instantly hit a wall as i need a way to log in. As I noticed the request library got fetching the login in screen. So I worked out that I can use selenium to log in as it required javascript. But a little use is that i confused how selenium will get the HTML as I still kept it on requests. But using the driver.page_source was able to transfer the HTML to the page to beautifulsoup so I thought I was in the clear but I still coundt find a way the select the element i noticed the code that beautifulsoup needed was there as I was using the search function on the anaconda prompt. But I spent a few hours on this issue. the main problem I noticed was the navigation of the HTML to find the element as the names of the classes and ids tend to be a random assortment of numbers and letters. so to was a maze trying to get selenium or beautifulsoup to select the element. Because I was getting not progress with this project I decided to move on to another project. Which the task was to make a web scraper that gets the information of the homepage of the items for sale. Also Similar to I project I did for myself. I started this project the next day as I was drained from the previous project. So I decided to start with beautifulsoup this time but I noticed I could get some of the information so i decided to bring in some selenium so I can navigate the javascript. With selenium, i handed over the HTML for the rest of the scraping with this I noticed that the class and id naming scheme made for sense so I could navigate more easily. But the more I tried to code I noticed I was forgetting the commands and the functions of the library so I had to reread on how to use beautifulsoup and watched the same videos for my first beautifulsoup project to get up to speed.