By Colin Keefe, IT Solutions Consulting
On Thursday May 9, it was my pleasure to walk through several new features in FileMaker Server 12, mainly concentrating on backups and remote container data storage. We wrapped at the 40 minute mark with plenty of time for questions – and boy, there were many. This review is my follow up to those questions. Some were answered during the presentation itself – for these I tried to provide a clearer explanation here. Some we didn’t get to or promised an answer later; for those I’ve provided explanations where I could.
There are one or two that I don’t have answers for currently, notably questions about the relative performance of secure vs. open storage over a WAN.
On the agenda for the presentation:
Changes in FileMaker Server 12’s Backup scheme and the new ability to store Container data externally offer significant improvements in data protection, performance and data delivery to the user.
In this session we explored how Progressive Backups work. We also discussed changes in the way Container data is delivered to the user through http streams, and we reviewed how all of this operates at the client level when building databases and navigating them as a user.
Topics Included:
- Full Backup functionality changes (or, what are hard links anyway?)
- Understanding, Configuring and restoring from Progressive Backups
- Understanding and implementing External Container Data storage
- How streaming and buffering works
- How to configure your Container field for external storage
- An explanation of Secure vs. Open Storage
Questions and Answers:
Q: So a hardlink links to ONE file, but what happens if the backed up hardlinked file somehow got corrupt, e.g. due to a faulty disk sector? Can you choose to schedule a daily FM 11 style backup to avoid this problem and then subsequent backups in the next 24 hours hardlink to that specific days’s full backup?
A. Yes. You can set up an old style set of 5 backup schedules, one for each day – M / TU / W / TH / F – and since they’re separate backup schedules, they would keep their own hardlink schema. In other words, files are only hard linked within a backup schedule, not between them.
Q: If I put my FileMaker database on a 3rd party outside server to run as secure SSL, what questions, such as who has access to the hardware, do you recommend asking to determine the best option? As many questions you can think of are very much appreciated.
A. The easiest way to determine suitability of an outside vendor for security purposes is to find out if they are SAS-70 compliant and, depending on your usage, HIPAA complaint. If they are, their website will say so. Note that this only offers assurances of security of the facilities and access to the hardware/OS. You need to confirm that your database itself has no design flaws that expose data to unauthorized usage.
Q: Can progressive backups be on an external or networked disk?
A. They can be on a locally mounted separate drive or a SAN. I don’t know if it’s technically impossible to write them to a networked disk but would strongly discourage it unless it’s a SAN.
Q: In the case where the FM12 container fields are used to store files externally… for each FULL individual backup, is the directory containing the external files also backed-up & duplicated? Thus generating a potentially very large storage need?
A. I punted on this one during the webinar. Basically the answer is that container data that hasn’t changed from one backup to the next IS hard linked.
Q: How do you suggest doing anti-virus checks and not destroying the data files?
A. The main thing you should do is provide exclusion rules for the AV software you’re using so that it omits key folders from the AV schedule. At a minimum I suggest excluding the Database and Backups folders. In rare cases of conflict with AV software I’ve had to exclude the temp folder because of overly aggressive AV checking on FileMaker’s scratch file.
FileMaker provides a best practices document for managing servers which we imagine will be updated for FMS 12 fairly soon. It’s on TechNet – which you should join if you’re not already a member. It’s free.
The FMS 11 version of this document is here:
https://fmdev.filemaker.com/docs/DOC-1123
Q: How do you recommend optimizing a system’s RAM since Filemaker 12 now uses half of a machine’s RAM?
A. I’m not sure if this is a FileMaker client or FMS question, but since this is a Server webinar, I’ll just say that RAM is just about the cheapest thing you can throw at a machine. Max it out. If that doesn’t get you to the minimum specs for FMS12 with memory to spare for the OS, then you should be considering a new server. There really isn’t anything wrong with upgrading your hardware every 3-5 years, and if you’re on the 5 year end of things, it’s going to be natural for the specs to no longer support the most recent versions of any software product.
There are a number of performance tuning recommendations in the best practices document linked above.
Q: Is there anyway to store a file somewhere other than on the server itself (e.g., a shortcut on the server pointing to another drive)?
A. I assume we’re talking about database files. If you’re talking about drives not physically attached to the server itself, the answer is only yes if we’re talking about a SAN, which has a very high throughput over fiber when configured properly.
The main reason for this is that FMS is VERY disk-intensive, and you want the path between memory and disk writes to have near zero latency. Optimally you want a very fast, local server grade drive subsystem, hopefully a RAID 5 or RAID 10, but at the very least, the best disk drive your money can buy.
Q: Do remote container files need to be stored on the same disk as FMS?
A. They will be stored wherever the actual database files are stored. You can’t put them somewhere else.
Q: Do remote container files get backed up as well?
A. Yes.
Q: To encrypt the files in container, we must have Filemaker Server 12 or can we do the same with Filemaker PRO 12, working as a Server (filetofile).
A. You can encrypt the files in a single user or peer-to-peer shared file. You cannot stream the data.
Q: Do remote containers support any RAW image formats?
A. They don’t. RAW image formats are data as exported from cameras – they’re proprietary to the camera make, and are “RAW” – what the camera captured without any post-processing. While various OS level image imports (like iPhoto on OS X) have RAW import support, the goal of an image management product like iPhoto is different from a database management product like FileMaker Pro.
Here’s a list of supported Container Field types:
http://www.filemaker.com/12help/html/add_view_data.4.34.html
Q: I already have a huge legacy file system with 50,000 files. How can I make that accessible through Filemaker? We need to be able to access it directly.
A. The need to access the container data directly is pretty much a showstopper for FMS 12. I would suggest using 360Works SuperContainer, which allows for image data hosted through an external web server – and from there you can configure the directories on that web server to be accessed the way you need. I would only point out that by accessing the data both ways, you face file management difficulties. What happens in the database when a source file is deleted? Who “owns” the data?
Here’s a side-by-side comparison of the relative advantages of FMS 12 and SuperContainer.
http://fmforums.com/forum/blog/62/entry-336-filemaker-12-and-supercontainer-a-comparison/
http://www.360works.com/supercontainer/
Q: Does file encryption affect database speed?
A. My guess is yes but I haven’t seen any documentation or metrics on this.
Q: Secure vs Open Storage – In large repositories, do you receive any type of noticeable performance hit for having these stored files encrypted?
A. This is the same question, with the same answer. I don’t know yet.
Q: Can you get FM to rename its stored copy of the file being imported? For example, customer is on a FM ‘job record No. 1234’ and imports an image called “my picture.jpg”. But we want it renamed and saved using the job number, e.g. ‘1234.jpg’? Does this type of functionality still require a plug-in?
A. Yes, you’re going to need a plugin for that.
Q: Is there any advantage to using webviewer versus container for purposes of accessing files or images through the database?
A. Yes, depending on your requirements. I would again point to 360Works SuperContainer (http://www.360works.com/supercontainer/).
Q: Are there any new ways of getting the meta data from a say a jpg?
A. To my knowledge, no – using GetAsText(containerfield) appears to work as before; there’s no real additional data. Remotely stored data is tagged as “remote:” but that’s about it.
Q: The difference between FM Server 12 and Filemaker PRO 12 working as a Server, and dealing with containers is with FM Server 12 we don’t need to wait to interact with the files, correct? The rest is equal?
A. To my knowledge, yes.
I strongly recommend not sharing peer-to-peer however. Over the long term there are too many variables that can be introduced to corrupt database files – non server-grade disk drives, ethernet plugs getting pulled on laptops, sleep mode activated, dependence on wireless signals…any and all of these things will happen over the life of a database ecosystem if it’s housed on a peer-to-peer network.
Q: If storing thumbnails, how do you determine when FM shows a thumbnail or the full resolution image?
A. My understanding is that when the “Generate and store thumbnails for images” bit is set in the Manage Containers dialog, then thumbnails are ALWAYS delivered to the client. The thumbnails themselves are generated and cached on the server.
You can get more granular with this using the GetThumbnail function to request a Thumbnail of a given height/width for a given field. So you could do, for example:
GetThumbnail( MyContainerField;
GetLayoutObjectAttribute(“myContainerFieldLayoutObject”,”width”);
GetLayoutObjectAttribute(“myContainerFieldLayoutObject”,”height”) )
Q: With thumbnails, if you use permanent storage and a file is updated or changed. Does it update the stored thumbnail, or leave the old one and create a new thumbnail creating an orphan?
A. I don’t know the answer to this! Thanks to each and every one of you for preparing me for my FileMaker 12 Certification Exam.
Q: If container storage directories on the server should be out-of-bounds to viewing and editing, is there any reason at all one might prefer to use Open container storage in a served environment?
A. To my mind, no. What hasn’t been answered (by me) is the question of whether or not the encryption affects performance, which would be a factor in this choice.
Q: Question on container field thumbnails being stored locally. Is FM smart enough to use the one thumbnail store (perm or temp) even if the same container field is on multiple layouts (and with multiple dimensions).
A. You guys are making my brain hurt. I don’t know.
Q: Great presentation! About container external storage, I would like to know if another user/application can modify the stored files on hard disk of the FM server ?
A. Yes, another application can modify Container Data files.
Since you can open container data in any number of applications, FileMaker Server has no control over whether you can edit the file in another application. What it CAN do is determine whether a file has been modified outside of its control – in which case you’ll get a message like this:
It does this for security reasons – if a hacker can replace/modify media on the server, then they can inject bad stuff into the data being streamed down to the client.
You can also query the validity of container contents using the new VerifyContainer calculation function.
The point I made during the webinar still applies here: leave the files alone. Remote Container Data storage is most definitely NOT meant as an OS-level image share/repository.
Q: Can the server be controlled (Backups, etc) from a remote location?? I can’t even change a global field!!!
A. Yes, the server can be controlled via the Server Admin Console, if the Server’s configured to accept Console requests from outside the physical machine.
I think your issue with editing a global field is unrelated, but I’ll just point out that global field behavior is a bit tricky when a file is hosted. Here’s a link to the appropriate Help File Topic:
http://help.filemaker.com/app/answers/detail/a_id/3604/~/global-fields%3A-an-overview
Q: “Compress this File” aka Archive is that still OK?
A. Yes. This seems like a good place to mention that the Save As Copy dialog now has a new option, as shown:
The “self-contained copy” gathers up any externally stored data into the new copy of the file, which offers a convenient way to build a consolidated file.
Q: If two different but identically named files are imported into the same container field on different records, does one overwrite the other or will FMS rename one of them accordingly?
A. The behavior here bears some explaining.
Open Storage, with default directory used:
When importing different, identically named files into the same container field on different records, FMS will rename the second file with a _1 incremented suffix.
When importing the same file into the same container field on different records, FMS will store only one copy of the file, and will keep that file until all records referencing it are deleted, or it’s cleared from all container fields referencing it.
If you want separate, explicit copies of the file, you can set the container folder to increment in the Options for the field, using the table’s unique ID, like this:
For example, in the image below:
- Two identically named, different files were imported into the Document_NoUniqueValue container field. The second file was appended with “_1”.
- Two identical files were imported into the Document_NoUniqueWitheIdenticalFiles container field. FileMaker Server kept one copy.
- Two identical files were imported into the Document_UniqueValue container field. Because the storage location for this field was specified by calculation (as noted above), each record has its own storage folder, “1” and “2”, so two copies of the file are kept.
All three of these fields are set to Open Storage. For Secure Storage, FMS keeps individual copies of files under its own schema.
The takeaway here is that FMS does manage all of this seamlessly and understands the difference between identical files and identically named files, so you shouldn’t have to worry about it.
Q: If you’re restoring from a backup…how do restore all the external container info also?
A. When you copy from the backup folder to the Databases folder, include the RC_Data folder that was backed up with the file, keeping the directory structure intact.
Q: Does remote storage work with IWP?
A. Yes. Streaming works too, but there are security considerations you should bear in mind.
Q: When using FileMaker with PHP, does a container content get streamed from the FileMaker engine or is it uploaded directly from the web server?
A. Well, interestingly, three separate components would handle this kind of thing:
- The Web Publishing Engine would handle the PHP
- The Database Engine would handle the database calls
- The TomCat HTTP Server embedded in the FMS 12 product would handle delivery of the container content.
As an aside, WPE has been completely rewritten from the ground up, and is significantly faster. This is not your Dad’s Oldsmobile, and I think you’ll be pleased.
Thank you to everyone who attended and for those who couldn’t make it, here’s the full video:
