Black Friday's Coming: Bronto's Ready November 13th, 2014 Erik Morse, Vice President of Software Development Erik Morse, Vice President of Software Development Erik Morse, Vice President of Software Development Author BioAs vice president of software development, Erik leads the effort to scale Oracle + Bronto’s technology organization in order to meet the increasing performance demands of the company’s rapidly growing roster of commerce-focused companies. Erik also leads Oracle + Bronto’s product development teams, bringing his lengthy SaaS experience to continually deliver innovative, yet easy-to-use functionality that helps retailers to maximize the impact of their marketing campaigns. Prior to joining Oracle + Bronto, Erik was the founding employee at Peopleclick, where he served for thirteen years as COO and CTO. During his tenure at Peopleclick, Morse was instrumental in building the technology organization and leading key product development initiatives, including the establishment of the company’s successful and rapidly scaling SaaS business model. Prior to Peopleclick, Erik served as a director of development for Seer Technologies, a leader in enterprise software infrastructure products. Previously, Erik held a key role at Digital Equipment Corporation (DEC) as a senior software engineer. Read More About Erik Erik Morse, Vice President of Software Development Each year Black Friday and Cyber Monday keep growing in importance. It’s critical that your marketing automation software function at peak levels during this time of feverish shopping. Don’t worry, we’re on it. Actually we’ve been on it. For months Bronto has been working to ensure you’ll be able to make even more sales this year and that the whole process will go as smoothly as possible. It was basically a two-fold mission. One – more capacity. Two – more speed. Here are a few of the ways we’ve done it: Event Broker Most important was the change we made back in the spring, when we switched our old event broker for a new, improved version. What’s an event broker and what does it do? The event broker sits at the center of our service-oriented architecture and manages the sending (or publishing) of events to other parts of the system that “subscribe” to those types of events. At any time, there may be tens of thousands of events per second. It’s the broker’s job to take those events and match them up with the subscriber that needs to know about it. For example, let’s say a contact on your company’s email list moves to North Carolina and changes his address on his contact form. Let’s also say your company has a North Carolina residents segment set up, to specifically market to that group. The broker publishes that event – the address change – to the subscriber – the North Carolina residents segment. That sort of activity happens over and over again, at every moment of the day. With our new event broker in place, we can handle more of those events and deal with them much faster. Segment Updates The previous example highlights another change: Thanks to our new event broker, we’re also now updating segments in real time. Previously, changes to a segment would be made en masse, often before a send. Now, as each change comes into the system, it’s immediately sent to the appropriate segment, which is altered accordingly. We also moved segments off our MySQL database servers onto Hadoop HBase data servers, a move that will have a big impact when Black Friday and Cyber Monday roll around. Sending and tracking, which make heavy use of MySQL, obviously spike on those days. By moving segments off MySQL, we lessen its load and free it up to perform sending and tracking functions more quickly. Workflows The key here was removing the occasional bottleneck that would sometimes slow down the workflows process. To do that, we added more HBase servers (20 in total), essentially spreading the workload out – also known as “scaling horizontally.” It’s a simple equation, really: The lower the workload for each server, the faster workflows can perform. Sending We know how vital sends are to you. So we’ve added more hardware to support this critical task. We’ve addressed a contention situation that sometimes resulted when inbound mail (returns from ISPs, bounced emails, etc.) would slow down outgoing messages. The method for throttled bulk sends has also been changed. Previously, they would be sent out at distinct intervals, causing spikes in the system. Now that load is spread more evenly over time. User Interface You’ll be working with our user interface quite a bit on Black Friday and Cyber Monday. That’s why we now have web servers dedicated solely to user interface activities. Keeping those servers free from the burden of other tasks will keep your interface running smoothly even during the most hectic times. We’ve also made changes to better handle those times when large amounts of data are being loaded onto our system, thus keeping pages in the application from loading slowly during that process. Storage/Servers There’s a lot going on in this area, but I’ll try to be brief. HBase – We mentioned earlier that we’ve significantly increased the number of Hadoop HBase servers, one of our primary methods of storing data. We’ve also improved our processes to be more efficient in how we use the allotted storage we have. MySQL – We’ve added more of these servers as well. And we’ve upgraded the software running on these databases to faster versions. Network Architecture – We replaced our old one entirely. Put simply: the new one makes everything faster, across the board. SANs – We’ve upgraded our Storage Area Networks (SANs) so that they operate faster and have more capacity. Why does that matter to you? Every time our database servers need to access our storage to retrieve some data, they need to interact with the SANs. Now we can handle more of those interactions and we can do it more quickly. Memcached – Caching servers let us operate on data in memory versus retrieving it from disk-based storage. It’s a much faster approach. We’ve also added a great deal of additional caching capacity. It’s a lot to digest, we know. But don’t worry. There won’t be a quiz. Instead, you can just focus on making Black Friday and Cyber Monday your most profitable ever. Are you ready? We certainly are.