Software Engineering Daily

How would you build a system for indexing and monitoring the entire Internet?

Start by breaking the Internet up into IP address ranges. Give each of those address ranges to servers distributed around the world. On each of those servers, iterate through your list of IP addresses, sending packets to them. Depending on what sorts of packets those IP addresses respond to, and what those responses are, you can build a map of the devices on the Internet: what is running on those devices, and what they respond to.

Qadium is a company that indexes and monitors devices on the Internet, to help organizations understand the devices that are within corporate networks. If you are a large corporation, Qadium can probably do a better job of figuring out your Internet footprint than you can.

Matt Kraning is the CTO of Qadium, and in today’s show he describes the process by which Qadium maps the Internet. Matt used to work on data infrastructure at DARPA, and has deployed Hadoop in Afghanistan–so the infrastructure of Qadium seems relatively manageable. Our data conversations in this episode spam from talking about Storm and Hadoop to Google BigQuery, BigTable, and DataFlow.

Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Every second your cloud servers are running, they are costing you money. Stop paying for idle cloud instances and VMs. Control the cost of your cloud with ParkMyCloud. ParkMyCloud automatically turns off cloud resources when you don’t need them. Whether you are on AWS, Azure, or Google Cloud, it’s easy to start saving money with ParkMyCloud. You sign up for ParkMyCloud, you connect to your cloud provider, and ParkMyCloud gives you a dashboard of all your resources–including their costs. From the dashboard, you can automatically schedule when your different cloud instances get turned on or off – saving you 65% or more. Additionally, you can manage databases, auto scaling groups and set up logical groups of servers to turn off during nights and weekends when you don’t need them–and you can see how much money you are saving. Go to parkmycloud.com/sedaily to get $100 in free credit for ParkMyCloud for SE Daily listeners. ParkMyCloud is used by McDonald’s, CapitalOne, and Fox, and saves customers tens of thousands of dollars every month. Go to parkmycloud.com/sedaily, and cut the cost of your cloud today.

Simplify continuous delivery with GoCD, the on-premise, open source, continuous delivery tool by ThoughtWorks. With GoCD, you can easily model complex deployment workflows using pipelines and visualize them end-to-end with the Value Stream Map. You get complete visibility into and control of your company’s deployments. At gocd.org/sedaily, find out how to bring continuous delivery to your teams. Say goodbye to deployment panic and hello to consistent, predictable deliveries. Visit gocd.org/sedaily to learn more about GoCD. Commercial support and enterprise add-ons, including disaster recovery, are available.

Incapsula can protect your API servers and microservices from responding to unwanted requests. To try Incapsula for yourself, go to incapsula.com/2017podcasts and get a free enterprise trial of Incapsula. Incapsula’s API gives you control over the security and performance of your application–whether you have a complex microservices architecture or a WordPress site, like Software Engineering Daily. Incapsula has a global network of over 30 data centers that optimize routing and cache your content. The same network of data centers that are filtering your content for attackers are operating as a CDN, and speeding up your application. To try Incapsula today, go to incapsula.com/2017podcasts and check it out. Thanks again, Incapsula.

Thanks to Symphono for sponsoring Software Engineering Daily. Symphono is a custom engineering shop where senior engineers tackle big tech challenges while learning from each other. Check it out at symphono.com/sedaily. Thanks to Symphono for being a sponsor of Software Engineering Daily for almost a year now. Your continued support allows us to deliver content to the listeners on a regular basis.