Wednesday, October 09, 2013

Introduction


Welcome to the LightWave Buyer's Beware Guide.

This blog is to air grievances with LightWave 3D and to point out a lot of utter stupidity that's has been finding its way into the software over the years. As someone who has used the software for over two decades, this saddens me to no end, as well as thoroughly pisses me off.
First, let's just get this straight...
I like LightWave.
 
But lately, it's feeling like it's got a bad drug problem. It shows some great progress, then follows that up by soiling itself somewhere. It put on some shiny new shoes, but hasn't changed its underwear in over a decade. It keeps wandering around aimlessly, like it doesn't know where to go. It used to innovate, but now it just follows the competition like a lost puppy, without appearing to even know why.

There used to be a time when LightWave could be considered a professional application. It worked without having to jump through hoops. It rendered beautifully, and it took effort to get a bad render. Bugs would get fixed in a timely manner, rather than when they seemed to fit within some abstract concept of what was appropriate for the version number. It was quick to learn, sensibly laid out, labeled with appropriately worded text labels instead of cryptic pictograms that needed text tool tips to pop up to tell you what a text label would already show you, and it had a nice progressive interface; tools were listed in related groups, with each additional tool being a refinement or variation of a previous one – move, followed by shear, which was move with fall-off, and then drag, which was another move with a different kind of fall-off, etc.

The most used features were right out there, in the open, ready to use. You start up the program, and you see a grid with a camera and a light, and the tools you need to manipulate those were right there on the interface. It was like walking into a studio. Keyboard shortcuts were likewise simple and easy to learn. Since this was not a text editor, there was no reason to complicate things with the unnecessary use of qualifier keys like CTRL or Shift for the most used features; a single keystroke was all that was needed, and that made it easy to remember. This was before monstrosities like Photoshop came about, with their hand-crippling concepts of keyboard “shortcuts”. Lesser used features were still easily accessible though they were placed in panels that would normally be closed and out of the way. Unlike many other applications, this was not littered with windows everywhere. It was clean.


There is a nice history of the LightWave interface at LightWave History wiki, where I grabbed some of the images here.

Over the years, LightWave has grown – a lot - and it had a few bumps along the way. LightWave 3.0 added a nice new, more colorful, interface, as well as many of the features that LightWave has been unfairly reviled for ever since, - namely bones and the deformation work flow. Many people that remember the days of LightWave 3.x will probably remember having to keep three different installations available at all times: 3.0, 3.1 and 3.5. Each had different features and different problems, so it was common to have to switch versions frequently.

LW 3.0 was the base of the 3.x series and it was pretty good. At this point, null objects were still actual files that had to be loaded from disc, that contained only a single point. Null objects were handled just like any other object, so you had to load them just the same. To load a null object, the entry in the scene file was
LoadObject nullobject
There were no filename extensions since this was on the Amiga, and the software on that was smart enough to know what format files were saved in. Your nullobject could be called anything, but since LW shipped with one in the content folder, you were set.

In LightWave 3.1, the scene file was changed. Null objects were now created internally, rather than loaded, through the use of a Add Null Object command. The entry in the scene was changed to:
AddNullObject nullobject

You could still load the old NullObject from disc, but on saving the scene, LightWave would check the object to see if it was a single point. If it wasn't, meaning it had more than a single point, the object would be loaded from disc. If it was a single point, then LightWave would write out the AddNullObject line instead.

This was one of the first main compatibility problems to arise in LightWave. Sure, there were problems going from 3 to 2 or from 2 to 1, but those were whole version steps there. (And oddly enough, some surfacing information from objects in version 2 did work in 1, even though you couldn't actually set them up in 1. Version 1 only allowed planar mapped Bump Maps, though cylindrical and spherical worked, but had to be set up in version 2.) But here, we're only a point-one away, and already, scenes from 3.1 would not load into 3.0 correctly. 3.0 didn't understand what AddNullObject meant, and it failed.

The solution was to force 3.1 to save that entry the old way, and to do that it had to be tricked. Since LightWave checked each object to see if it was a null, it would write out AddNullObject when the scene was saved. But, if you had two points (or any number, really; as long as no polygons were in the object, it would not render, which is still true today.) in the object, then LightWave saw this as a regular object during scene saving. It would still act like a null in Layout – it would not render, not cast shadows, or do anything else that a null object would not do – but when it came time to save the scene, this was an object like any other. LightWave 3.1 would save it out with a LoadObject entry.

Now you could directly load any 3.1 scene into 3.0, and vice-versa. Why would anyone want to do that?

Bugs.
Not quite the bugs I had in mind.

Both versions had, what we started to call, “bug sets.” Some features didn't work in 3.0, while other features didn't work in 3.1. So we would have to figure out which set of bugs we could work with, and that meant having to freely switch between 3.0 and 3.1. This meant having to copy files around in the Video Toaster directory each time, though, since at the time, these were just two executables, I could get away with just renaming them. Even the object structure changed slightly, and there was a utility you could get that would 'fix' 3.1 objects so they could load into 3.0. This also worked for 3.5 objects, which could have a few more textures and bumps than were present in 3.1.

This could be considered the birth of the dreaded workaround. Ask any LightWaver that's been using the software for a year or more, and they'll tell you about workarounds. Techniques that have been developed to work around flaws in the software. Over the years, the number of workarounds has grown considerably, to the point that LightWave is now known as much for its workarounds as it is for its rendering, if not more so.

This has never been a good thing, and it's only gotten worse over time.

Now, it's expected that as software develops, it gets new features, and new problems. It shouldn't be, but the reality is, people make mistakes. But there is a difference between inadvertent problems, and intentional ones. Bugs are things that are not intended, usually due to typos in coding, lapses in logic, or just stuff that doesn't want to work in unexpected situations. Fixing those is a daily routine for programmers.

Leaving them in, in the hopes that no one will notice, is a bad thing. Leaving them in, when they're obviously wrong and highly visible, and have been complained about by users, well, that's just stupid, insulting, and reeks of arrogance. Here, I'm talking about things that aren't rendering the way they should, cause unnecessary problems, are just inconsistent with the rest of the program, or are just blazingly wrong. Features that work in the reverse of what they should fall into this category. Frequently these can be nailed down to simple errors like a negative value that should be positive, or a documentation error, that describes a feature wrong, or even backwards. LightWave 5.5's Natural Shaders were guilty of this, where their Fresnel settings worked in exactly the opposite way they were described in the manual.

Which one was correct? I still don't know. I don't know if the documentation for that has ever been fixed to match the behavior, but I do know the features have not been altered to match the documentation. That would be bad since it would mean reversing how they worked, and that would lead to more compatibility issues again. In this case, the features in question were the incidence angles, and the interface wasn't clear about how those angles were utilized, so either way was actually valid. They just didn't match the description given in the manual. But if you never read the manual, you could figure these out yourself. Reading the manual would have actually added to the learning curve here. It did for me.

A common complaint in the LightWave forums and mailing lists, going back to the late 90's, is one of interface consistency. In the early days, LightWave's interfaces were very consistent; they had the same look and feel, as though they belonged together. This held true for 3.x, continued with version 4, which was the first to be released on the PC and other systems, and then with 5.0. Plug-ins were also new as of 4.0, and this is where the interface consistency issue started to pop up. Rather than make the interfaces of their plug-ins look like the rest of the software, many 3rd party developers chose to take a shortcut and use their own designs, or just have the OS handle it. The result was a lot of plug-ins that just didn't feel like they belonged, and when open, made the interface feel like it was something that was just hacked together by kids over a weekend.

Probably the first real plug-in for LightWave.

It was no longer looking like a serious, unified, professional application. Sure, other applications were suffering the same fate; anyone remember Kai's Power Tools for Photoshop, and how well those were received? That doesn't mean they're examples to be followed.

Possibly the most horrid  interfaces in history.

This was also at a time when LightWave was new to the PC and Mac worlds, and it still had “Amiga cooties.” A time where 3D Studio was so vastly pirated it was hard to run into a PC user that didn't have a copy or two. A time where LightWave really needed to up its game to be taken seriously. And it was, for a while, mainly in TV production, where it ruled the VFX land for shows like Babylon 5, Star Trek, and many others in the mid to late 90's.

However, LightWave 4.0 was actually a really weak offering. It was flaky and would crash if you looked at it wrong. Its new tools, like Metaform, were as likely to create a garbled mess of lines as they were to create smooth organic shapes, and, contrary to many claims, this was not a sign of a crack, but of fragmented memory. Use it long enough, and this would happen. This didn't help its acceptance much.

This was the first time it became preferable to use the old version; in this case, 3.5. The studio I was at used 4.0 for about two weeks, on and off. We went back to 3.5 to get work done, and only used 4.0 to occasionally use some of the new modeling tools, which didn't work half the time.

Then version 5 came out. This was pretty sturdy and is what version 4 should have been. Same interface, a few more plug-ins, but it worked.

Then things started getting weird.

LightWave 5.5 came out, was a paid upgrade, and had a whole new interface. That's something you would normally expect for a point-zero release, not a point-five release That is not to say it was bad. Version 5.5 added some very useful tools, fixed the spaghetti bug with Metaform, added the Natural Shaders, for landscape texturing, a couple volumetric systems; HyperVoxels and Steamer, and even included a lite version of Particle Storm, making this version a bit more of a serious tool for studios.  With these new plugins, and more third party plugins, came a slew of various interface styles, which started to make the software look and feel like it was a hack job.  Particle Storm, for example, did not look like it belonged...

Particle Storm interface.
 
 LW 5.6 fixed a few issues, and was a pretty solid workhorse from 1998 to 2000. If LW 5 is what 4 or 4.x should have been, then, LW 5.5 is what 5.0 should have been. New problems started to appear. Volumes were not visible through transparent surfaces, so to get that to work, you had to add a plug-in to those surfaces. The same went for shadows; a HyperVoxel would not cast a shadow on a surface unless that surface had a shadow plug-in applied. And with a limit of four plug-in slots available, that made things very tight.


LightWave 6 came out in 2000, and once again, the interface changed, and that basic layout is still with us in version 11. It also added the ability to edit the interface for both Layout and Modeler. The initial configuration was very simple, particularly for Layout, which only had three tabs across the top. With all the new floating panels you could have, this wasn't really a hindrance, since you could get to everything with just those three tabs.



The plug-in architecture was finally starting to mature, and the surfacing system underwent a major overhaul. Gone were the limits of the four texture slots. Motion Channels could finally be edited separately. Object layers, which were previously only a storage space in Modeler, were now a full blown object feature, and could be loaded into Layout, with each being treated like a separate object. The Gradient that was a staple of HyperVoxels was now a full blown surface attribute. A new, interactive particle system was added, that worked right there in Layout. Multiple morph shapes could be defined right within an object itself. You could layer unlimited textures over one another. This meant that older assets would not work 100% in the new version; texture velocity was removed, but replaced with the far more versatile ability to animate textures in any direction, rotation or scale, but older objects could still be load, as could many older scenes. Anyone would have to agree that was a good trade up. But with the new interface and what was supposed to be a complete rewrite of the base code, came problems. Again, this is to be expected with a point-zero release, but this sucker was buggy.

Not to be confused with buggy suckers.

LightWave six also added some very powerful tools, notably, a built in radiosity solution and a new polygon subdivision feature. It was also the first ever application to incorporate HDRI (and I suspect this was also its last innovation). It also replaced the old HyperVoxels 1.0 with a new, more powerful 3.0 version (version 2.0 was a separately available commercial plug-in) and replaced Steamer with more powerful volumetric options for its lights. The old Steamer and HyperVoxels plug-ins would still work, and were included for backward compatibility, so nothing was taken away here (except their interfaces, which were disabled now).

NewTek started really getting crazy with names at this point, many of which I've never heard another user mention in any discussions because the names really didn't matter; they didn't show up in the interface anywhere. PAVLOV was the name given to the expression engine (I think), but everyone just called them expressions. Intelligentities was another name that was only used in sales material or in a copyright section of the manuals, and referred to the endomorphs or morph maps that objects could now contain. 

Within a year, LightWave 6.5 came out, and many people thought that was what 6.0 should have been. It expanded the interfaces to include more tabs on Layout, and sorted everything very smartly in both applications. These became know as the John Gross configs, after the owner of one of the popular VFX houses at the time, Digital Muse, who created it based on the needs of production. These new configs were the standard for the next few years.



With 6.x, several new issues reared their ugly heads. Communication was a big one. There were many new plug-in classes available, but many did not talk to one another. You could add a motion modifier to something, but it wouldn't work if you had IK on. Particles didn't find deforming surfaces reliably. This is when we started to find out that that big rewrite from the ground-up, didn't get as close to the ground as we were told.

LightWave started attracting a lot of new 3rd party plug-ins, which again, brought more alien interfaces into the mix. There didn't seem to be any design guide for anyone to use. It also didn't help that many of the really useful plug-ins were Japanese in origin, so even their labels were not very helpful. At the very least, there was something called X-panels that they could use, but this became reviled by the users and programmers alike because it was too limiting. Apparently, even something simple like getting stuff aligned neatly in a panel was outside of the scope of this thing, so we also kept getting plug-in interfaces that just looked sloppy.


This trend continued in 7.0, which came out very shortly after 6.5, and added a lite version of Worley's fur plug-in, Sasquatch, which again, had its own interface style, like all of Worley's plug-ins. LightWave 7.0 also added a new scene editing tool called Spreadsheet. This thing was completely different in both look and feel from anything else in LightWave. It had some good ideas, but the work flow was atrocious. To call it clunky is being very kind. It would also kill a render farm in a blink of an eye because the render nodes didn't understand its settings in scene files. To this day, I'm not sure if that bug ever got fixed. I do know that it stopped a lot of people from ever using it again.

LightWave 7.5 added a few new goodies, fixed a few issues, but generally, wasn't that big of an upgrade. It added a few new modeling tools, a few OpenGL enhancements, and a few bug fixes, but it was really a lackluster release. Then again there were other things going on behind closed doors at the time, which basically equated to us being lucky we got anything at all. This was 2002, and everyone else was serious about Maya and Max and XSI, which were now really leading the way. LightWave was really starting to stagnate.

Things really soured for the release of 8.0. That gave us a new darker, solid color scheme, so gone was the weird brushed pastel look. A lot of people thought it looked a lot like 5.5 or 5.6 again, which wasn't necessarily a bad thing. This version also added new tools for bones, and a very powerful IK supplement called IK Booster. Larry Shultz frequently showed its capabilities at presentations, and it was extremely impressive, but this tool never really took off, mainly because it was hard to find and used inconsistent methodology. It also didn't help matters that NewTek also did not seem to know anything about this tool (or suite of tools as it really was), and they never did anything to promote it.

There were also a lot of bugs in 8.0, and these were serious bugs; broken textures, textures that had bad errors in them, texture layers that would have their orders reversed if you copied them, color channels that turned to greyscale, numerous HyperVoxel problems, etc. All these really hurt us in production. And to top it off, NewTek decided to just change the interface again, for no good reason. Gone, were the production-proven layouts of tools that long time users were used to. Gone, were the quick and simple, single-keystroke shortcuts for the most used functions. Now we had configs that had stuff like Undo use a two key combination, while Redo had a single keystroke. Why the lesser used function was easier to use than the more used one is beyond me. So, not only were we getting hit with bug after bug, but we had to struggle with an alien layout of tools. And by alien, I mean, alien. These didn't even feel like they were designed by anyone that actually used the software. Older users tended to hate it.

Not quite that alien.

LightWave 8.0 was quickly followed by 8.0.1, which fixed many of the severe render error bugs, like the broken textures. Unfortunately, this was already broken long enough (just four months) that it became the expected behavior by several new users who were using it in production, so this fix broke those assets. This was not the first time this happened, and it would not be the last. It was however, caught and fixed promptly.

Four more versions of 8 came out that year, and LightWave 8.5 was a fairly solid tool again, though it was really starting to showing its age. Radiosity was still really slow, though the Backdrop Only version was acceptable in most cases. Caustics were still useless. HyperVoxels got some fixes, and volumes finally worked correctly with fog for the first time. A new antialiasing mode, PLD, was added, giving users much more control over the quality of their renders, allowing up to 35 AA passes. There was also a 64bit version of 8.5.

In 2006, LightWave 9.0 came out. This phase became a bit of a game changer. With this, the polygon subdivision system got a good overhaul, implementing something called APS, or Adaptive Polygon Subdivision (they prefer to say the P is for pixel, but that doesn't make any sense since no pixels are ever subdivided, but polygons sure are), which allowed the user to control how subpatches were divided, based on time, distance, angle, etc.. It also added the concept of camera plug-ins, and added the powerful Advanced Camera, where the user could actually use geometry as a camera plane, or create spherical or panoramic cameras, or pretty much anything he could think of.

This initial release also offered what is probably be biggest, most powerful feature yet – nodes. At this point, it was only for surfaces, Volumetric Lights, and deformations, but for a point-zero release, this was a very well thought out feature, and it filled in many old requests, dating back to the 90's, mainly being able to texture and envelope everything. Every value in nodes was not only something you could change over time, you could use a texture to control it, and have an envelope or texture control that, etc.


It also added the ability to replace the built-in shading models that LightWave had used up until now, namely, Lambert for diffuse and Phong for specularity. There were four new Shader inputs; Diffuse, Specular, Reflection, and Refraction, and each could be controlled, not only in how intense they were or what shading algorithm was used, but also their color, so for the first time, you could easily have, say, a yellow surface with red reflections and blue transparency. The drawback with nodes was also their selling feature. They were powerful and you could do pretty much anything, and that made them harder to learn and longer to set up. I know few people who use them on a regular basis, preferring to stick with the faster layers system for surfacing. Unfortunately, this was also the last version of nodes that was so well thought out.

LightWave 9.0 was probably the biggest, most powerful update in LightWave's history at this point, even beating out the advancements of versions 2 (over 400 new features) and 3 (bones and flares and shadow maps etc.). And it added one other thing; another set of configs, called “Production Style.” It seems enough people complained about the new layout in 8 that NewTek went ahead and made another preset for the configs. So, if this new one is the production configuration, what does that make the one that was added in 8? I can only assume that if one is the production configuration, then the other one is not a production configuration. In which case, why is it included, and more importantly, why is it the default?

Then 9.2 came along. With that came another addition to nodes – the Material Nodes – which offered newer, and simpler ways of doing things like glass or metal, or even subsurface scattering effects. Well, simpler, if you knew the secret, which I'll cover in another post. This version also drastically changed how we would render. Previously, antialiasing was done by rendering the frame multiple times, each with a slight offset and those images – called passes – were then averaged together, and we could do up to 35 of these. In 9.0, new cameras were added, and these new cameras offered something the old, Classic Camera didn't, and this release opened that up. Now each pixel could be rendered with a different time than the others, and this allowed features like a smooth, photo-real motion blur, and much better depth of field. Adaptive Sampling, which, up until now, was really just an evil to be avoided, was now a feature you didn't want to render without. The old Classic Camera was still present and still worked on the multiple passes method, but the new Perspective Camera, which was a full-tilt ray-tracer, let us use as many passes as we wanted, and as many samples per pass as we wanted. Radiosity was greatly sped up as well. A render that would take 8 hours with Monte Carlo radiosity would only take two hours now. And the improvements here were not done yet.

Version 9.3 to 9.6 really improved a lot of things. By the time 9.6 came out that same radiosity render would be done in about four minutes! LightWave was a severe multi-threading race horse, topping out at 16 threads in 9.6, dynamically reallocating them as each segment of a frame finished. Unlike the old camera, where one segment would finish and that CPU would just sit idle until the others finished, every CPU (or core) would be 100% utilized until the frame was done. Again, this meant renders were faster. This also meant that blurred reflections, which were also now fully controllable with nodes, were finally a usable feature.

Unfortunately, we still didn't see much improvement in other areas. Particles, which were added in 2000, were not touched much. Dynamics in general were still very clunky to use. HyperVoxels, aside from the bug fixes in 8.x, didn't get much attention, though they did finally get multiple blending modes that haven't been seen since the HV 2 version, and you could finally have more than 65,000 of them. And if that wonderful photo-real blur sounded to good to be true, well, it was partially. It didn't work with volumetrics; those would not blur during a frame unless you used multiple passes, and that meant losing the speed and quality of photo real blur. It also didn't like anything being deformed by bones, so any kind of creature or character animation would cause it to become unacceptably slow, and in this case it was far faster to use multiple blur passes. Animated textures would not blur, and many envelopes also did not. It now became a chore to keep track of what worked with what, and some animators literally had a checklist of these things written on paper.
 And then LightWave died. The official word was there was not going to be a LightWave 10. There would be no more bug fixes, so reporting anything new would just be a waste of time. Something else, called Core, was going to be the new big thing. Some people were wetting themselves with anticipation, while older, more experienced users were far more skeptical. Why upgrade to something that's never been used, and was going to be so completely different and have to learn it from scratch, when we can already do that by switching to Maya, or XSI, or anything else that's already in use? Some people did just that. Plus, it's a new application, so it's going to be a long time before it was ready for production, let alone even actually be used in production. No one really believed it would be ready in the time frame we were told. Those that did were deluded, and tended to be from the pant-wetting group mentioned above. LightWave was dead, at the ripe old age of 19.

A year later, in 2010, a zombie apocalypse happened, and LightWave walked the earth again. LightWave 10, the version we were told would never exist, came out of nowhere. With it came VPR, a new interactive renderer like F-Prime, but built into the interface. We also got new color space features and the first of a bunch of motion capture tools. Damn. So those bugs I've been getting hit with in 9.6 for the past few years could have been reported and fixed after all? Unfortunately, LightWave 10 was horrendously slow compared to 9.6. The next release, 10.1, sped things up again, including a major speed up in volumetric lights. It also broke any assets that used the Gradient node, which I'll cover later.

Shortly after the release of 10.0, another update to the 9 series, version 9.6.1 came out, upping the number of possible threads to 64, as in 10, as well as fixing a few more things.

Then in early 2012, LightWave 11 was released. Sure it had a lot of good stuff, but the way it was handled was actually insulting. This began a pattern of taking features away from users, and replacing them with different features that did not fill the gaps. It also made the cardinal sin – replacing the render engine. Of all the things that needed to be brought up to date, the render engine was pretty low on the list. As has been seen during the 9.x phase, it really got sped up, but now we had a new one that used something new called Unified Sampling, and in every case I've seen, it rendered slower, needed more samples, and still rendered noisier than 9.6 did. It also took away the control that animators tend to need, like the quality controls for lights, and the shading controls for nodes. It stupidly grouped these into two values on the Render Panel. Now, if you needed to increase the quality of just one light, you were forced to raise it for every light in the scene. More on this later.

Blurred reflections were also changed. Instead of adding a new reflection blurring method, like through a new reflection shader, they changed it internally, everywhere. Now blurred reflections are more streaky looking, like what you get with wet surfaces. When these are used, everything looks wet. If you needed the old method, well, you're screwed. Go back to 9.6 or 10. More on this later as well.

On the bright side, we got some wonderful goodies like a new built-in instancing system, Bullet dynamics for hard body simulations, and flocking (though nothing to update the 12 year old particle system). These were all fine tuned a bit with three minor updates that year.


In January, 2013, 11.5 came out. This finally brought some of the new features up to what they should have been in 11.0; adding soft bodies to Bullet, more control over flocking, and some work flow enhancements. VPR could now render DOF and motion blur. Modeler finally got a few new tools though some of them were basically duplicates of existing tools. A new rigging tool was added to Modeler. Sampling controls in Layout were still absent though. Of the 11 phases, I would say that 11.5 (and the later 11.5.1) was the least offensive of the bunch.

I could write a book on all the problems, but who would want to buy that? Instead, I'll waste my precious free time here, in the hopes that this will give certain people an idea of how LightWave 3D is viewed in production, when no one is looking. I've seen first hand how attitudes change when the NewTek people show up at studios; they're all polite and friendly and cordial, but as soon as the NewTek presence has left the building, the gloves come off and the swearing starts again.

Occasionally, you'll see some scores at the bottom of posts. These should be self explanatory, but one does deserve an explanation. The FFU score is short for Feature|Fuck-U, and is scored on either thumbs or middle fingers. Middle fingers mean the “feature” is more of an insult in its implementation while thumbs up mean it's a good thing.