I seem to have reached this limit while editing my answer to:

I've inherited 200K lines of spaghetti code -- what now?

This is a bit frustrating, as the editor didn't warn me, so I typed away and submitted... and it truncated the end of the text AND the links...

So now I'll gladly rewrite this quickly, and make sections more concise to fit everything in, but:

  • What exactly is the limit?
  • Maybe the editor should warn and block submission, like the comment section does.

Though I agree such long posts probably do not happen too often. Sorry. got interested in this question more and more :)


Update 2012/07/11:

Steps to reproduce:

  • The "test-case answer" was created by simply identifying where things went wrong in my answer to the question linked above and I narrowed it down to between revisions 12 and 13.
  • So I created a question with content from revision 12 (+disclaimer a the top about testing),
  • then edited it with content from revision 13 (but with the link table of revision 12, to get the untrucated ones in there as well, as the revision 13 was obviously damaged),
  • the submission went through just fine and without warning, and the content was indeed truncated again.

Update 2012/07/11 - continued:

Crap, it appears the revision log of my test-case doesn't show all the steps, and doesn't show the messages I had left in the "change logs". Probably because I did them without a big interval between them, and when that's the case the SE systems seem to merge them in one changeset (probably so we don't hit the 10 edits => CW rule when editing answers rapidly while they're new).

So, that's annoying here for looking at the test-case (maybe SE gods have a way to see full logs), but the steps I gave above should be enough to reproduce it anyway.

Update by Shog9

Reproduced, definitely a bug, written up on MSO.

share
    
As a follow up, I did manage to find this question on Meta Stack Overflow. It specifically mentions question validation, though. –  Thomas Owens Jul 9 '12 at 11:42
    
@ThomasOwens: it mentions that some action was taken, but it doesn't look that way. –  haylem Jul 9 '12 at 11:44
    
I don't want to post a 30k character question just to test it out, but if that feature was implemented and hasn't since been removed, it appears to only apply to questions and not answers. Given that, I'd suggest posting on MSO, referencing this question and the other question on MSO. –  Thomas Owens Jul 9 '12 at 11:46
    
@ThomasOwens: Ok. Thanks a lot, Thomas. Will give this a shot (probably tomorrow, a bit rushing now). –  haylem Jul 9 '12 at 11:50
    
@haylem Given your tendency to write extra long posts, we'd love it if you'd be interested in writing for our community blog. If you are, visit the blog chat room and talk with Dynamic or World Engineer... –  Yannis Rizos Jul 11 '12 at 7:15
2  
@YannisRizos: I could be interested, though I wouldn't know what to write about. Whereas when I have a question to answer, I may have something to ramble about. That being said I remember seeing on the programmers blog that "non constructive questions make for good blog posts" or something similar, so I guess I could pick one of these up :). Anyways, I'm flattered, and will try to catch up with them on the chat when I can later tonight or in coming days. –  haylem Jul 11 '12 at 10:49
    
@haylem You can start as a proofreader, if you can spare the time. I'll be off next week (vacation time, yay!), but afterwards I can help find some highly upvoted deleted questions for you to look at and see if you're interested in converting them to blog posts. –  Yannis Rizos Jul 11 '12 at 10:55
add comment

3 Answers

up vote 6 down vote accepted

Unless things have changed (and given the status-declined tag, it doesn't appear that way), an answer is limited to 30,000 characters.

The explanation given is that posts that exceed 30,000 characters are rare and might indicate other problems with the question. Like the FAQ says, if a book is more suited to answer the question, then it's not an appropriate question for a Stack Exchange site. 30,000 characters is (by my estimate) 12 pages of double-spaced 12 point Times New Roman text - if a question actually requires that much text, it probably has problems. For example, it might be asking multiple questions that should be broken out into independent questions or it might be overly broad (one of the reasons for closing as a non-constructive question).

You're right that truncation is not the answer, though. It's not very user-friendly when content gets chopped off without warning.

share
    
about truncation: it's indeed painful... I was just throwing stuff in... and now am struggling to remember. It's generally a good thing as you end up rewording things in a more efficient phrasing, BUT it's VERY demotivating! Especially when you already spent something like 40 minutes typing it in, and that people will thing you're leaving a botched anwswer with broken links and text. –  haylem Jul 6 '12 at 20:55
    
Thanks for the quick answer Thomas. Anything special to do to push for this "warning" feature-request? –  haylem Jul 7 '12 at 10:02
    
@haylem It's tagged as feature-request here, but it applies to all Stack Exchange sites. I would see if it has been posted on Meta Stack Overflow and what it's status is. I did a really quick search and didn't see it. I believe the SE community team and/or the developers watch the feature-request tag on all of the sites, though, so someone should see it. –  Thomas Owens Jul 7 '12 at 11:57
add comment

I can't reproduce this; trying to submit several chapters from Alice's Adventures in Wonderland produces,

Oops! Your answer couldn't be submitted because:

  • body is limited to 30000 characters; you entered 34491

Submitting an edit using the default inline editor behaves a little differently - I got a little pop-up reading,

body is limited to 30000 characters; you entered 34701

Opening the editor in a separate tab reverts to the submission failing with an error:

Oops! Your edit couldn't be submitted because:

  • body is limited to 30000 characters; you entered 34837

Even here, the text wasn't truncated - the submission was flat-out rejected. It may be that some combination of formatting or link markup specific to your answer allowed it to bypass the first check and be accepted (but truncated); if you can reproduce this, please post details.


haylem managed to reproduce this quite effectively! I've written up a full bug report on MSO: Max length check ignores markup, results in truncation of long, formatted answers

share
    
That was in an edit indeed. Might be that I got a warning after the fact, once changes were already sent. But it definitely did submit the edit, so changes were lost, and the content was truncated (end of text and links table missing). –  haylem Jul 9 '12 at 16:53
    
I'm leaving for the night now, but I'll try later to recreate a post in this Meta questions with exactly the same series of edits and see what happens (hopefully that should be the same on Meta and main? I wouldn't want to post test-questions and answers on main) –  haylem Jul 9 '12 at 16:57
    
Sorry, couldn't get back to it until now... So, if I look at the history, Revision 12 was still OK with 24,385 chars (according to emacs. YMMV, for instance depending on how you count newlines, the link table, etc...). Revision 13 is said to have added 5076 characters, which would bring us up to 29,461 chars (though emacs reports 29405...). And it is truncated in the middle of the links. –  haylem Jul 11 '12 at 0:24
    
the note in the revision log says added 3146 characters in body, so I assume that excludes the link table. So that's still odd, as that would mean an even lower count, so we're not exactly at the limit. There must be a difference in the way we count them. –  haylem Jul 11 '12 at 0:26
    
My bad, I have 597 lines... had forgotten that emacs' char counter would probably not count \n chars. So it would appear I did manage to submit at least something more or less exactly on 30K. Still need to determine how it went through: did I automagically screw up something and cut it myself (not that I remember... and it happened again in revision 14 when I added the note about having reached the limit: the lower part of the text gets shortened again), or did it do its on its own without warning me? Will try in an answer here now. –  haylem Jul 11 '12 at 0:27
    
tada: use case provided in the other answer... I think I can guess where the bug lies: it probably has to do with how you count chars, and the fact that you don't consider the link table in the counting. So it doesn't look like I have reached 30K and you don't warn me, BUT when the answer gets submitted, it then truncates the content to something around 30K still (so, in the middle of the links). –  haylem Jul 11 '12 at 0:34
    
well, my last assumption was wrong (even with body > 30K, it let me submit and truncated the post). Bug is definitely present, but I don't know the cause. Maybe some other elements of the body aren't counted properly? newlines, images, quotes, code blocks, etc...? Let me know if I can be of more help. Look at the revision log of the answer below to see the test case and reproduce if necessary. –  haylem Jul 11 '12 at 0:41
    
Thanks @haylem, that's perfect. I've reproduced the problem, and written it up in full on MSO! –  Shog9 Jul 11 '12 at 1:23
    
Cool, glad I could help. Let us know how it goes. –  haylem Jul 11 '12 at 1:57
add comment

THIS IS A TEST CASE, please do not screw with this answer.

See the comments under in Shog9's answer, and other answers for details.


Foreword

This whole thing sounds like a daunting task indeed, and there's a lot of ground to cover. So I'm humbly going to suggest what I think could be a rather comprehensive guide to use for your team, with pointers to appropriate tools (and alternatives) and appropriate reading or educational material to share.

Please do keep in mind that these are guidelines, and that as such they are meant to adapted, adopted, or dropped based on circumstances. Furthermore, dumping all this on a team at once would have no effect, except if they are ready to listen. It's more likely that you should try to cherry-pick some elements that would give you the best bang for your sweat, and introduce them slowly, one at a time.

Note also that not all of this is easy to apply to Visual or Graphical Programming Systems, like G2. For more specific details on how to deal with these, see the Addendum section at the bottom.


Executive Summary for the Impatient

  • Define a rigid project structure, with:
    • project templates,
    • coding conventions,
    • familiar build systems,
    • and sets of usage guidelines for your infrastructure and tools.
  • Install a good SCM and make sure they know how to use it.
  • Point them to good IDEs for their technology, and make sure they know how to use them.
  • Implement code quality checkers and automatic reporting in the build system.
  • Couple the build system to continuous integration and continuous inspection systems.
  • With the help of the above, identify code quality "hotspots" and refactor.

Now for the long version... Caution, brace yourselves!


Rigidity is (Often) Good

This is a rather controversial opinion, as rigidity is often seen as a force working against you and slowing you down. It's true for some phases of some projects. But once you see rigidity as a structure, a framework that takes away the guesswork, it greatly reduces the amount of wasted time and effort. Make rigidity work for you, not against you.

Another way to think of this, is that Rigidity = Process / Procedure. In a chemical plant, you are likely to have a lot of manuals and procedure for how to do things. i.e. A) Turn this valve B) Check this gauge C) adjust this temperature. The reason for this is quality and safety. You want everyone to do it the same way, ever time. This reduces bad outcomes (injuries, poor quality production, etc.) and maximizes good outcomes.

Software development needs good process and procedures for exactly the same reasons.

Rigidity of the Project Structure

If each project comes with its own structure, you are lost and need to pick up from scratch every time you look at it, and the same applies to each newcomer. You don't want this in a professional software engineering shop, and you don't want this in a research lab either.

Rigidity of the Build Systems

As mentioned above, if each project looks different, there's a good chance they also build differently. A project's build shouldn't require too much research or too much guesswork. In general, you want to be able to do the canonical thing and not need to worry about specifics: configure; make install, ant, mvn install, etc...

A quick README at the root to point to things that differ, but that's all there should be (in an ideal world).

Plus, this also greatly facilitates other parts of your build infrastructure, namely:

  • [continuous integration][1],
  • [continuous inspection][2].

It also helps to ensure that all projects are built to the same level of quality, but re-using the same build system for all of them and making it evolve over the time. Not only do you keep it (and all your projects) up to date, you also make it stricter over time, and more efficient at reporting potential mistakes and enhancements. Do no reinvent the wheel for each project, and reuse what you have already done.

Recommended Reading:

  • [Continuous Integration: Improving Software Quality and Reducing Risk][3] (Duval, Matyas, Glover, 2007)

Rigidity in the Choice of Programming Languages

You probably can't expect, especially in a research environment, to have all teams (and even less individual developers) use the same language and technology stack. However, you can identify a set of "officially supported" languages and frameworks, and encourage their use. The rest of other languages, without a good rationale, shouldn't be permitted beyond prototyping.

It is essential to keep your build system simple, and the maintenance and breadth of required skills to a bare minimum, a core of technologies and tools.

Rigidity of the Coding Conventions and Guidelines

Coding conventions and guidelines are what allow you to develop both an identity as a team, and a shared lingo. You don't want to err into terra incognita every time you open a source file.

There's no use trying to enforce non-sensical rules that will make things harder or to forbid things to the extent that commits would be refused based on a single violation. However is takes away a lot of the whining and of the thinking if you identify a clear, concise set of ground rules that nobody should break under no circumstances, and a set of recommended rules that are advised to be followed.

I am fairly aggressive when it comes to coding conventions, some even say nazi (without wanting to offend anyone with the evocation), because I do believe in having a lingua franca and a recognizable style for my team. When crap code gets checked-in, it stands out like a cold sore on the face of an hollywood star, which helps you to identify that a quick review and action are required. In fact, I've sometimes gone as far as to advocate the use of pre-commit hooks to reject commits if they do not satisfy some common rules. As mentioned before, it shouldn't be overly crazy and get too much in the way, especially as you try to introduce these measures. But it may be well-worth it if you spend so much time reviewing and dealing with crap code that you can't work on real issues.

Some languages enforce some rules by design. Java was meant to reduce the amount of dull crap you can write with it (though no doubt it can be done, as evidenced here and on SO), for instance. Python's block structure by indentation is another idea in this sense. Or the Go programming language with its gofmt tool, which completely takes away any styling work - and ego!! - out of coding effort: if you run it before every commit, things are sure to be always looking fine for everybody.

Be sure to make it so that critical code gore cannot slip through. Code conventions, continuous integration and continuous inspection, and pair programming and code reviews are your best weapon against this demon.

Plus, as you'll see below, code is documentation, and that's another area where your conventions should encourage proper readability and clarity.

Rigidity of the Documentation

Documentation goes hand in hand with code. Code itself can be documentation. But there must be clear-cut instructions on how to build things, how to use things, and how to maintain things.

Using a single point of control for documentation (like a WikiWiki or DMS) is a good thing. Create separates spaces for projects, separate spaces for more random banter and experimentation. And make sure that each of these spaces reuses a set of common rules, and that people take care of following them when they edit it.

In fact, most of the instructions that apply to code and tooling here apply to documentation as well.

Rigidity in Code Comments

Code comments, as mentioned above, are also documentation. Developers like to express their feelings about their code (mostly pride and frustration, if you ask me). So it's not unusual for them to express these in no uncertain terms in comments (or even code), when a more formal piece of text could have conveyed the same meaning with less expletives or drama. It's OK to let a few slip through for fun and historical reasons: it's also part of developing a team culture. But it's very important that everybody knows what is acceptable and what isn't, and that comment noise is just that: noise.

Rigidity in Commit Logs

Commits logs are not this annoying thing of the usage lifecycle of an SCM that you just need to skip to get home on time or get on with the next task, or to catch up with the buddies who left for lunch. They matter, and, like (most) good wine, the more time passes the more value they have. So make sure they are done right. I'm always flabbergasted when I see co-workers writing one-liners for giant commits, or for non-obvious things.

All commits are done for a reason, and that reason may not be clearly expressed in the one line of code you added and the one line of commit log you entered. There's more to it than that. Each line of code has a story, and a history. The diffs can tell its history, but you have to write its story.

Why did you need to update this line? Because the interface changed.

Why did the interface changed? Because the library that provides it was updated.

Why was this library updated? Because it's a dependency on another library that we needed to implement feature X.

And what's feature X? All about it is in TASK_KEY_HERE.

Git actually gets this right in that it is more geared towards providing good logs than any other SCM. Though it's not my SCM of choice, and not necessarily the best one for your lab either; but it gets this right. It lets you provide a short log, and long log. Leave the general update to the shortlog, with the reference task IDs to link to your issue tracker (yes, you need one), and expand in the long log. Write the changeset's story.

For crying out loud, if you can do it on a blog, you can do it in a log. It's the same origin for (We)Blogs, after all: just keeping track of things.

Really ask yourself the question:

If I were searching for something about this change later, would this log answer my questions?

Documentation and Code, and Projects as a Whole, Are ALIVE

You need to keep them in sync, otherwise they do not form that symbiotic entity anymore. That's why it works wonders when you have:

  • clear commits logs in your SCM, with links to task IDs in your issue tracker,
  • where this tracker's tickets themselves link to the changesets in your SCM, and possibly to the builds in your continuous integration system,
  • and a documentation system that links to all of these.

Code and documentation need to be cohesive.

Rigidity in Testing

Any new code shall come with (at least) unit tests.

Any refactored legacy code shall come with unit tests.

Period.

Of course, these tests need to actually test something valuable, and to not be just a waste ot time and energy. They need to be well written and commented, just like any other code you check in. They are documentation as well, and they help to outline the contract of your code. Especially if you use [Test Driven Development][4]. But even if you don't, you need them for your peace of mind. They are your safety net for the future (for maintenance, for future enhancements) and your antibiotic against normal code rot.

And of course, you should go further and have [integration tests][5], and [regression tests][6] for each reproducible bug you fix.

Rigidity in the Use of the Tools

Sure, it's OK for the occasional developer/scientist to want to try some new static checker on the source, generate some graph or model using another, or implement a new module using a DSL. But it's best if there's a canonical set of tools that all team members are expected to know about and to use.

I regard it as generally OK to recommend a default working environment with these tools, but to let each developer use their IDE or editor of choice, as long as they are productive AND do not require regular assistance to adjust to your general infrastructure AND do not modify the common areas (code, build system, documentation...) in ways that affect other developers. If that's not the case, then it's fair to enforce that they fallback to your defaults.

Note: Of course, some flexibility is good. Letting someone occasionally use a shortcut, a quick-n-dirty approach, or a favorite pet tool because it gets the job done is fine... But never let it become a habit, and don't let this snippet of code or prototype become the actual codebase to support.


Team Spirit Matters

Develop a Sense of Pride in Your Codebase

  • Develop a sense of Pride in Code
    • Use wallboards
      • leader board for a continuous integration game
      • wallboards for issue management and defect counting
    • Use an [issue tracker][7] / [bug tracker][8]

Avoid Blame Games

  • DO use Continuous Integration / Continuous Inspection games: it fosters good-mannered and [productive competition][9].
  • DO keep track defects: it's just good house-keeping.
  • DO identifying root causes: it's just future-proofing processes.
  • BUT DO NOT [assign blame][10]: it's counter productive.

It's About the Code, Not About the Developers

The whole point to make developers be conscious of the quality of their code, but to see it as a detached entity and not as an extension of themselves (and react badly when a part of this extension is criticized. Encourage [ego-less programming][11] for a healty workplace but do rely on ego for motivation.


From Scientist to Programmer

You can't expect people who do not value and take pride in code to produce good code. They need to discover how valuable (and fun) it can be, for this property to emerge. Sheer professionalism and desire to do good is not enough: good code needs passion. So you need to turn your scientists into programmers (in the large sense).

Someone argued in comments that after spending 10 to 20 years on a project and its code, anyone would feel attachment to the code. Maybe I'm wrong but I assume they are proud of the code's outcomes and of the work and legacy it represents, not of the code itself and of the act of writing it.

From experience, most researchers regard coding as a necessity, or at best as a fun distraction, except for the ones who are already pretty verse and attacted by it. They just want it to work. The ones who have an interest in programming are a lot easier to persuade of adopting best practices and switching technologies. You need to get them there.


Code Maintenance is Part of Research Work

Nobody wants to read a crappy research paper. They are proof-read, refined, rewritten, resent for re-approval countless times until they reach this final state that's deemed good enough for publication. The same applies to a thesis. And the same applies for a codebase!

You want to make it clear that constant refactoring and refreshing of a codebase is what prevents code rot and technical debt, and what facilitates future re-use and adaptation of the work for other projects.


Why All This??!

In the end, why do we need all the above? For the Holy Grail: code quality. Or is it quality code...?

All of the above aims at driving your team towards this goal. Some aspects of it does it by genuinely wanting them do it themselves (which is much better) and others by slightly taking them by the hand (but that's how you educate people and develop habits).

But how do you know if you have found the Holy Grail, and not some cheap knock-off (which might make you turn to dust quickly, which is unpleasant)?

Quality is Measurable

Not always quantitatively, but it is measurable. As mentioned above, you need to develop a sense of pride in your team(s), and showing progress and good results is key. Measure code quality at point T, and show progress between intervals. Show how it matters. Do retrospectives to reflect on what has been done, and how it made things better or worse.

There are great tools out there for continuous inspection. [Sonar][12] being one of them, quite popular in the Java world, but it can adapt to other technologies; and there are many others. Keep your code under the microscope and look for these pesky annoying bugs and microbes.


But What if My Code is Already Crap??

Of course, all of the above is fun and cute like a trip to Never Land, but it's not that easy to do when you already have (a big pile of steamy and smelly) crap code, and possibly a team reluctant to change.

Here's the secret: you need to start somewhere.

Personal anecdote: In our current project, we are working with a codebase that originally was more than 650,000 lines of Java code, more than 200,000 lines of JSPs, more than 40,000 lines of JavaScript, and more than 400 MBs of binary dependencies on external projects and libraries.

Today, after about 18 months, we have 500,000 lines of (MOSTLY CLEAN) Java code, around 150,000 lines of JSPs, and still about 38,000 lines of JavaScript, and our dependencies are down to barely more than 100MBs (and these dependencies are not in our SCM anymore!).

How did we do it? We just did all of the above, or we try to.

It's a huge team effort, but we slowly inject new regulations and new tools that help us to monitor the heart-rate of our product, while we hastily slash away the fat of the crap code and useless dependencies we can find. We didn't stop all development to do that. We have occasional periods of relative peace and quiet where we are more or less free to go crazy on the codebase and tear it apart, but most of the time we just do it all by defaulting to sort of a "review and refactor" mode every chance we get: when things build, over lunch, during team bug fixing sessions, when Friday afternoons get drowsy...

We did have a few big construction sites... Switching our build system from a giant Ant build of more than 8500 lines of code to a multi-module Maven build was one of them. We now have clear-cut modules (or at least it's already a lot better than before, and we still have big plans for the future), automatic dependency management (which allows for easy maintenance and updates, and allowed to remove lots of them), and faster builds that are easier to get started with and to reproduce on demand, and to integrate with code quality tools.

Injecting some "utility tool-belts" into the codebase, even though we were trying to reduce dependencies, was another: Google Guava and Apache Commons can help you code slim down to a much smaller size, and reduce surface for bugs in your code a lot.

Persuading our IT department that maybe using the tools we use today (JIRA, Fisheye, Crucible, Confluence, Jenkins) was better than the ones in place. We still need to deal with a few ones we despise (I'm looking at you, QC, Sharepoint and SupportWorks), but it's still been a huge improvement, and we believe there's still room for improvement.

And every day, there's a trickle of between one to dozens of commits that deal only with fixing and refactoring things. We do occasionally break stuff (remember, you need unit tests kids, and better write them before you refactor stuff away), but overall the benefit for our morale AND for the product has been enormous. We get there one fraction of a code quality percentage at a time. And it's fun to see it increase!!!

It's indeed important to note that every once in a while, rigidity needs to be shaken to make room for new and better things. In my anecdote, our IT department is partly right in trying to impose some things on us, and wrong for others. Or maybe they used to be right. The point is you need to prove that they are indeed better, and will boost your productivity. Trial-runs and prototypes are here for this.


The Super-Secret Incremental Spaghetti Code Refactoring Cycle for Awesome Quality

       +-----------------+                     +-----------------+
       |                 |                     |                 |
       |  A N A L Y Z E  +-------------------->| I D E N T I F Y |
       |                 |                     |                 |
       +-----------------+                     +---------+-------+
                ^                                        |
                |                                        v
       +--------+--------+                     +-----------------+
       |                 |                     |                 |
       |    C L E A N    +-------------------->|      F I X      |
       |                 |                     |                 |
       +-----------------+                     +-----------------+

Once you have some quality tools at your toolbelt:

  1. Analyze your code with code quality checkers.

    Linters, static analyzers, or what have you.

  2. Identify your critical hotspots AND low-hanging fruits.

    Violations have severity levels, and large classes with a large number of high-severity ones are a big red-flag: as such, they appear as "hot-spots" on radiator/heat-map types of views.

  3. Fix the hotspots first.

    It maximizes your impact in a short time-frame as they have the highest-business value. Ideally, critical violations should dealt with as soon as they appear, as they are potential security vulnerabilities or crash causes, and present a high risk of inducing a liability (and in your case, bad performance for the lab).

  4. Fix the low-level violations with automated codebase sweeps.

    It reduces the noise-to-signal ratio so you are be able to see significant violations on your radar as they appear. There's often a large army of minor violations at first if they were never taken care of and your codebase was left loose in the wild. They do not present a real "risk", but they impair the code's readability and maintainability. Fix them either as you meet them while working on a task, or by do large cleaning quests with automated code sweeps if possible. Do be careful with large auto-sweeps if you don't have a good test suite and integration system, and make sure to agree with co-workers on the right time to run them to minimze the annoyance.

  5. Repeat until you are satisfied.

    Which, ideally, you should never be, if this is still an active product: it will keep evolving.

Quick Tips for Good House-Keeping

  • When in hotfix-mode, based on a customer support request, it's usually a best practice to NOT go around fixing other issues, as you might introduce new ones unwillingly. Go at it SEAL-style: get in, kill the bug, get out, and ship your patch. It's a surgical and tactical strike.

  • But for all other cases, if you open a file, make it your duty to:

    • imperatively: review it (take notes, file issue reports),
    • maybe: clean it (style cleanups and minor violations),
    • ideally: refactor it (re-organize large sections and their neigbors).

Just don't get sidetracked into spending a week from file to file and ending up with a massive changeset of thousands of fixes spanning multiple features and modules, or it makes future tracking difficult. One issue in code = 1 ticket in your tracker. Sometimes, a changeset can impact multiple tickets; but if it happens to often, then you're probably doing something wrong.


Addendum: Managing Visual Programming Environments

The Walled Gardens of Bespoke Programming Systems

Multiple programming systems, like the OP's G2, are different beasts...

  • No Source "Code"

    Often they do not give you access to a textual representation of your source "code": it might be stored in a proprietary binary format, or maybe it does store things in text format but hides them away from you. Bespoke graphical programming systems are actually not uncommon in research labs, as they simplify the automation of repetitive data processing workflows.

  • No Tooling

    Aside from their own, that is. You are often constrained by their programming environment, their own debugger, their own interpreter, their own documentation tools and formats. They are walled gardens, except if they eventually capture the interest of someone motivated enough to reverse engineer their formats and builds external tools... If the license permits it.

  • Lack of Documentation

    Quite often, these are niche programming systems, which are used in fairly closed environments. People who use them sign NDAs a dime a dozen and never speak about what they do, and programming communities are them are rare. So resources are scarce. You're stuck with your official reference, and that's it.

The ironic (and often frustrating) bit is that all the things these systems do could obviously be achieved by using mainstream and general purpose programming languages, and quite probqbly more efficiently. But it requires a deeper knowledge of programming, whereas you can't expect your biologist, chemist or physiscit (to name a few) to know enough about programming, and even less to have the time (and desire) to implement (and maintain) complex systems, that may or may not be long-lived. For the same reason we use DSLs, we have these bespoke programming systems.

Personal Anecdote 2: Actually, I worked on one of these myself... I didn't do the link with the OP's request, but my the project was a set of inter-connected large pieces of data-processing and data-storage software (primarily for bio-informatics research, healthcare and cosmetics, but also for business intelligence, or any domain implying the tracking of large volumes of research data of any kind and the preparation of data-processing workflows and ETLs). One of these software was, quite simply, a visual IDE that used the usual bells and whistles: drag and drop interfaces, versioned project workspaces (using text and XML files for metadata storage), lots of pluggable drivers to heterogeneous datasources, and a visual canvas to design pipelines to process data from N datasources and in the end generate M transformed outputs, and possible shiny visualizations and complex (and interactive) online reports. Your typical bespoke visual programming system, suffering from a bit of NIH syndrome under the pretense of designing a system adapted to the users' needs.

And, as you would expect, it's a nice system, quite flexible for its needs though sometimes a bit over-the-top so that you wonder "why not use command-line tools instead?", and unfortunately always leading in medium-sized teams working on large projects to a lot of different people using it with different "best" practices.

Great, We're Doomed! - What Do We Do About It?

Well, in the end, all of the above still holds. If you cannot extract most of the programming from this system to use more mainstream tools and languages, you "just" need to adapt it to the constraints of your system.

In the end, you can almost always version things, even with the most constrained and walled environment. Most often than not, these systems still come with their own versioning (which is unfortunately often rather basics, and just offers to revert to previous versions without much visibility, just keeping previous snapshots). It's not exactly using differential changesets like your SCM of choice might, and it's probably not suited for multiple users submitting changes simultaneously.

But still. If they do provide such a functionality, maybe your solution is to follow our beloved guidelines, and to transpore them to this programming system!! Implement your tests within the platform itself, and use external tools and background jobs to set up regular backups. Sure, it's a hack job and definitely to up to the standard of what is common for "normal" programming, but the idea is to adapt to the system while trying to maintain a semblance of professional software development process.

If the storage system is a database, it probably exposes export functionalities, or can be backed-up at the file-system level. If it's using a custom binary format, maybe you can simply try to version it with a VCS that has good support for binary data. You won't have fine-grained control, but at least you'll have your back sort of covered against catastrophes and have a certain degree of disaster recovery compliance.

WARNING: this section is still in progress. Check again later... again. Sorry.


NOW ADDING CRAP CONTENT HERE TO PUSH THE BODY's char count higher than 30K, and ensure we don't have (body < 30K) but a ((body + links) > 30K).

EDIT: Well, just a note to clarify... my assumption was wrong, it just fails to warn and truncates. The bug must lie somewhere else. Sorry. And good luck with it :)

loremp ipsum whatever and so on... loremp ipsum whate

share
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .