Constantly Start Over
Never Rewrite
Constantly Rewrite
@mattjmorrison
Constantly Start Over
What are the Best Projects?
Green Field
Green Field == Starting Over
Why?
So Many Reasons
- No Technical Debt
- No Legacy Code
- Chance to Learn from Your Mistakes
- Chance to Choose
- New Languages
- New Platforms
- New Frameworks
- Updated Versions
The Unfortunate Reality
Is That Most of Us Work In a…
Brown Field
The Trick is Turning
Your Brown Field Green
Not as Simple as Watering
What Makes a Brown Field Brown?
Many Reasons, But Mostly
Something That I Call…
The Monolith
What is The Monolith?
It’s One Large Code Base
It Does All the Things
Monolith Advantages
- All the Things in One Place
- Simple to Get Started
- Easy to Reuse Code
- Simple Networking & Hardware Needs
- Simple Deployment
Neutral Fact
Many Frameworks Have Monolithic Tendencies
- Rails
- Django
- «Insert Full Stack Framework Here»
Caveat - Rails Engines & Django Apps - Use These
Monolith Disadvantages
- It’s Big, Scary and Causes Nightmares
- Difficult to Fit in Your Head
- Merge Conflicts
- Slow Tests (because there are LOTS of them, right?)
- Gaps in Coverage (100k LOC @ 96% = 4k Untested LOC) *
- Difficult to Leverage Open Source Community
- Easy to Introduce Unnecessary Coupling
- You Don’t Know You’re Building One Until It’s Too Late
* MS DOS 1.0 was 4k Lines of Assembly in 1980
What Does the Monolith Have to do With a Brown Field?
Wikipedia: Expansion or redevelopment of (brownfields) may be complicated by environmental contaminations.
The Monolith is a Contaminated Environment
coders anonymous
Hi, my name is Matt
and I’ve built Monoliths
Let’s Travel into the Past
It is Now 2009
Chesley B. Sullenberger III Saves 155 Lives by Crash Landing in the Hudson River
And We Begin Building a New Web Based System
Fast Forward to 2012
Robotic Rover Curiosity successfully lands in Gale Crater on Mars
And Our Web Based System has Grown Significantly
- Customer Management
- Customer Usage Reports
- Manage Users
- Authentication
- Authorization
- Display Customer Data
- Generate Reports for Customer Data
- Create New Business
- Work Flow Management
- Integrate with Geo Coding Services
- Integrate with Multiple Property Valuation Services
- Integrate with Multiple Reporting Agencies
- Manages Attaching Documents
- Migrating Legacy Data into Database
- Exporting Data into Multiple 3rd Party Systems
- More…
It Does Too Much
- Everyone Fears the Monolith
- Everyone Tries to Avoid Changing It
- Bugs are Explained Away
- Bugs are Shrugged Off
Time for a Rewrite
Never Rewrite
An Entire System
What value is there in a Rewrite?
- Cleaner Code
- Better Techniques
- Fewer Bugs
- Better Test Coverage
- Newer, more up-to-date Technologies
- You know everything
- Green Field, Baby!
What Business Value is there in a Rewrite?
- Long Time to Market
- Fewer Features
- Training Users on a New System
- Second System Syndrome
- Existing Production Support
So Then, What Do You Do?
Make Smaller Pieces
If You Do Not Have One Giant Code Base
You Will Not Have One Giant Rewrite
And What About the Monolith?
Iterate
As opportunities arise…
- Add Tests & Refactor
- Extract Services
- Extract Libraries
- Extract Applications
- Integrate the Old with the New
Let’s Take a Look at Some Examples
Extract Services
Example
What is a Service?
A Thing in the Cloud that You Can Use
- Some Code Deployed Out on the Web Somewhere
- Exposes an API to Allow Manipulation or Retrieval of Data
- Clients Need to Know Very Little About It
New Vendor Integration
- UI To Order Reports From Vendor
- UI To Display Report Results
- Use Report Results in Existing New Business Application
Our Choices
Add to the Monolith?
Boo, Brown Field
- or -
Start Over?
Yay, Green Field
Start Over
- New Project
- New Code base
- New Database
- New Web System with an API
- Green Field, Baby!
Roadblock #1
Authentication
Easy with a Monolith
Not So Easy with Multiple Servers
Solution?
Start Over
Green Field, Baby!
- New Project
- New Code Base
- New Database
- New Web System with an API
- Replace Authentication in Monolith with Calls to the New Authentication API
- Migrate Existing User Data to New Database
Roadblock #2
Customers
Again, Easy with a Monolith
And Again, Not So Easy with Multiple Servers
Remember that Unnecessary Coupling Thing?
Solution?
Start Over
Green Field, Baby!
- New Project
- New Code Base
- New Database
- New Web System with an API
- Remove Customer Management from the Monolith, Add API Calls
- Migrate Existing Customer Data to New Database
Where are we now?
Where We Ended Up
Pros
- Smaller Pieces with Fewer Responsibilities
- Pieces are More Loosely Coupled
- Pieces Can be Updated in Isolation
- Continuous Reevaluation of Technology
- Fewer Technology Restrictions
- Lots More Green Field Work
Cons
- Slow to Get Started
- No Out-of-the Box Solution
- More Complex (or at least more) Deployments
- No Single Place to Find All Answers
- Still a Bit of Work in the Brown Field
- Performance - DB Join is Now an HTTP Request
A Few More Things About Starting Over
It Sounds Daunting, But the More You Do It
- You Get Better
- You Get Faster
- You Get to Apply Things that You’ve Learned
Extract Libraries
Example
What is a Library?
A Portable Bundle of Code
- Can Be Included Into Other Systems or Libraries
- Provides Reusable Bits of Code
How Do We Share Common Code?
- Monolith
- Authentication
- Customer
- Vendor Integration
Duplicate Code
Extract to Libraries
Things to Consider
- Semver
- Tag Versions in Source Control
- Internal Hosting
- Dependency Management
- Open Source
Open Source
Just By Trying to Name Your Open Source Library You May Find That It Already Exists
Open Source
django-pretty-times
Translates a Timestamp into:
- Just Now
- _ Seconds Ago / In _ Seconds
- _ Minutes Ago / In _ Minutes / A Minute Ago / In a Minute
- _ Days Ago / In _ Days / Yesterday / Tomorrow
- _ Weeks Ago / In _ Weeks / Last Week / Next Week
- _ Months Ago / In _ Months / Last Month / Next Month
- _ Years Ago / In _ Years / Last Year / Next Year
django-pretty-times
- Very Small Library
- Very Specific Function
- Open Source on Github
- Available on PyPI
- 9 Community Contributions
- 5 Community Contributors
Extract Applications
Example
What is an Application?
A Vertical Slice of a System
- Has A UI and/or API
- Has Some Logic
- Has Persistence
- Is Pluggable
Authorization
Each System Needs Permissions
And a Way to Maintain Them
The Monolith Can Do It
Time to Extract
Green Field, Baby!
Hold On…
Have I Contradicted Myself?
Extracting
≠
Rewriting
Or Is It?
When I Say “Extract”
I Do Mean “Rewrite”
But
On Such a Small Scale That it Does Not Count
Extracting is more of a Refactoring than it is a Rewriting
Also, since it is a Refactoring…
For The Love of All that is Good and Holy
Get some tests around the code AS IT IS prior to extracting anything!
Ideal Situation
- Thorough Tests Around the Code-To-Be Extracted
- Delete Said Code, Not Tests
- Build Out Green Field Project (Baby!)* to Take Extracted Code’s Place
- Drop in Newly Extracted Code
- Tests Still Pass
* With Tests, of Course!
Bad Situation
- No or Very Few Tests Around the Code-To-Be Extracted
- Delete Said Code, and Tests (If They Exist)
- Build Out Green Field Project (Baby!) to Take Deleted Code’s Place
- Drop in Newly Extracted Code
I Have Tried This, It Does Not Work
The Rewrite to Never Do
Is the One Called The Rewrite
Integrate the Old with the New
Do Not Rewrite, Integrate
Legacy Desktop Application
The Web is the Future
We Want to Leverage What We Already Have
Legacy Desktop Application
- ~ 30 Years Old
- Windows Desktop Application
- PC COBOL & Flexus SP2
- PC COBOL does not speak HTTP
First Bit of Integration
Legacy Application Does Some Calculations
We Want to Leverage That from the Web
We Do Not Want Possible Inconsistencies
How Do We Integrate?
- Install the Legacy Software to a New Internal Server
- Add a Command Line Interface
- Wrap a Web Service Around It (Green Field, Baby!)
- Profit
Second Bit of Integration
Legacy Application Has Some Data
We Want that Data to be Pushed in Real Time to the Web
How Do We Integrate?
COBOL Does Not Speak HTTP, We Start There
We Made COBOL Speak HTTP
- New Web Service to Accept Legacy Data (Green Field, Baby!)
- C# COM Component to Talk to New Web Service (Green Field, Baby!)
- Active X Control to Pass Data to COM Component
- Flexus SP2 with Active X Control Add-On
- PC COBOL Program to Pass Data to SP2
- Lots and Lots and Lots of Integration Tests
What Do We Have Now?
- Our Legacy System Can Communicate with New Systems
- Our New Systems Can Communicate with our Legacy System
- We Can Incrementally Add, Enhance and Replace Features
Important Saftey Tip
Use Adapters
What is an Adapter?
Code that Sits Between
- Your Code and Other Code
- Your Services and Other Services
- Your Old Systems and Your New Systems
Why Use an Adapter?
Because it Allows Your Code to
- Define the API
- Define the Data Input/Output Formats
- Define the Field Names
- Define the Authentication
- Define the Authorization
- Not Care About the Rest of the World
Do Not Allow External Variables to Compromise Your Design
Examples
- Command Line Interface to Legacy Desktop System
- Web Service that Wrapped Command Line Interface
- Cobol Program to Talk to C#
- C# Code to Talk to Web Service
- Web Service that Accepts Data from C#
- Web Service to Talk to Vendor(s)
- Libraries that Talk to Web Services
How To Get Started
Start Small
Start Simple
Extract One Small Thing at a Time
Extracting Everything at Once is The
Rewrite that Should Never Be Done
Simplest First
- Test & Refactor
- Extract a Library
- Extract an Application
- Extract a Service
- Integrate with an Old System
Tools
(In Order of Importance)
- Source Control
- Tests
- Environment Isolation
- Automated
- Database Migrations
- Dependency Management
- Deployment
- Server Configuration
Need Buy-In?
Back Yourself into a Corner
Remove the Choice and Require it
The Trick is Turning
Your Brown Field Green
Started With A Brown Field Project
Added A Whole Bunch of Small Green Field Projects
Currently at
55
Web Services, Libraries, Applications and Other Small Systems
The Monolith Has Not Changed All That Much
But It Is Starting to Look a Lot More Green
Thanks!