The Rest Of The Story

My special interest is computers. Let's talk geek here.
User avatar
yogi
Posts: 9978
Joined: 14 Feb 2015, 21:49

Re: The Rest Of The Story

Post by yogi »

The Bible reminds me of Linux. There is a kernel of truth in every one of them, but then the Free and Open Source authorities pervert it every which way they can. I think Windows might equate to the New Testament, specifically the Book of Revelations. Apocalypse Now! Fortunately there is no ambiguity about the origins and intent of operating systems. They can even be useful on occasion.
User avatar
Kellemora
Guardian Angel
Guardian Angel
Posts: 7494
Joined: 16 Feb 2015, 17:54

Re: The Rest Of The Story

Post by Kellemora »

How many Bible Thumpers does it take to change a light bulb.
None! They prefer to remain in the dark, hi hi.
User avatar
yogi
Posts: 9978
Joined: 14 Feb 2015, 21:49

Re: The Rest Of The Story

Post by yogi »

I recall the first "born again" Christian I met. He worked at Motorola and was quite a likeable fellow, except for his obsession about being born again. We became friends and I invited him and his wife to our home for a small get together. It was a pleasant evening, but the inevitable happened. He and his wife were on a mission and felt we needed to be converted. He kept going on and on about seeing The Light. So, I asked him how does one know when they see The Light. His reply was quick and simple. "You are blinded by it." Yes, indeed. That explained it all. :lmao3:
User avatar
Kellemora
Guardian Angel
Guardian Angel
Posts: 7494
Joined: 16 Feb 2015, 17:54

Re: The Rest Of The Story

Post by Kellemora »

Ha ha. I don't get along to well with those types either, hi hi.

I've never understood why Christians chose a Device of EXECUTION as their primary logo, and plop one atop all their buildings. And the Catholics still have Him hanging on the cross.
That was NOT the important event of Christianity! It was actually the lowest point. Nothing to celebrate!
User avatar
yogi
Posts: 9978
Joined: 14 Feb 2015, 21:49

Re: The Rest Of The Story

Post by yogi »

The good nuns in my grammar school explained that very phenomena. I've been thinking hard to try and recall the rationale, but am drawing a blank. I have a feeling it's something related to the concept that Christ died for our sins. It seems that everybody was a sinner until Jesus came down from heaven and washed them all away. It was a clean start, but how long did THAT last? LOL Once more I'm reminded of a software analogy. Windows works at it's best the minute it is installed fresh. From the next minute on it's all downhill.
User avatar
Kellemora
Guardian Angel
Guardian Angel
Posts: 7494
Joined: 16 Feb 2015, 17:54

Re: The Rest Of The Story

Post by Kellemora »

True on all counts Yogi!
User avatar
yogi
Posts: 9978
Joined: 14 Feb 2015, 21:49

Re: The Rest Of The Story

Post by yogi »

Just so you know, I'm composing today's responses from inside a Linux Ubuntu virtual box. I tried to do an update but it could only partially install all the things available. That caused me to trash the original VM and create a new one. It was then that I realized that I didn't do a full install, but only the minimum install. I did that because they recommend 25GB for Ubuntu to work properly. Well, I keep my virtual machines to 10GB and all that explains why a full update is not necessary. I know you can't tell the difference between today's writing and other times. That's because Windows is the host. LOL
User avatar
Kellemora
Guardian Angel
Guardian Angel
Posts: 7494
Joined: 16 Feb 2015, 17:54

Re: The Rest Of The Story

Post by Kellemora »

The few times I tried VM or VB, I gave it like 100 to 150 gigs. But then too I needed a lot of file storage for each one.

Speaking of which, I have another question.
RSync normally runs really fast, once all the files are in place and in sync.

It will run super slow if you are moving something from the same hard drive to the same hard drive.
That is understandable.

I moved an entire 110 gig file folder from a 120 gig drive to a 500 gig drive. Needed the extra space to merge another 130 gig file to it. Gotta be careful doing things like that if they are going in the same main folder, else everything will get deleted except the new file, hi hi.
The first move went super slow as expected since it was a new copy not yet synced.

OK, the files are now in the same folder, so I can safely run my backup of the main file to the backup drive which already had the first main file on it. This second hard drive is 2 terrabytes.

Perhaps I should explain it this way first. I went through all of my old IDE drives and copied everything to a 250 gig HD.
As I copied each IDE drive, I first copied the files as they were in the folders they were in. Once I had all of them loaded I compared each into a new file, and deleted duplicate copies. Then I spent the time to put away the files and documents in new more meaningful folders.
Then I created a new master folder for all other folders to go into more new folders that made more sense, hi hi.
First I had to move the 110 gig file to a 500 gig hard drive, so I could move a 130 gig file I also already sorted out to it.
It took forever to merge the 110 gig file with the 130 gig file because they were on the same drive.

Once I was pleased with the file progress the way I wanted it, the 500 gig drive will become my daily working drive.
The 2 terrabyte drive strictly for a backup from the 500 gig drive.
Naturally the first copy took forever, which is OK.

When I work on a file I saved on the 120 gig drive, then ran RSync to copy it to a backup drive. It ran like lightning.
Same with the 130 gig file to its backup drive.

OK, now everything is on a 500 gig SATA drive, this is my daily working drive.
And backed up to the 2 terrabyte SATA USB drive.
Right now I'm up to 270 gigs of data on both the 500 gig drive and on the 2 terrabyte drive synced.
If I don't make a change to any folder or file and run RSync, it scrolls through all the documents in less than 5 minutes, then exits with zero like it is supposed to. Sometimes as fast as 3 minutes.
Now, if I add another document or two, or change the data on a few documents and run RSync.
It will run fast up to about 80% then come to a dead crawl and sometimes take up to a half hour or longer.
It will slowly get up to 98% and then take another 20 minutes to complete from there.
It reads the HD in order as you can see on the display, and although I made no changes in the later files where it has bogged down. I'm wondering why this is happening.

Have I made my master file too big? Should I have kept certain files folders separate from the main folder?
Or does it have to do with how much memory my computer has?

When RSync is running, it naturally reads every file to see if something changed, but this goes fairly quick, and when it gets to one that did change, it deletes it and rewrites the changed copy, just like it is supposed to, then it goes back to reading more files looking for a change.
Once it has got through the ones that changed and proceeded to read the rest of the drive, as it nears the end this is when it suddenly bogs down. And it gets slower and slower.
But as I said, if I make no changes to any files and run RSync it takes only 3 to 5 minutes before it exits.

Don't know if this will help, but here goes.
Folder Number One, the ONLY folder on the 500 gig drive is named Master-File-Server.
Inside that folder are folders named: Business, Family, Property, and Reference.
And naturally, inside each of those folders are hundreds of sub-folders, some nested pretty deep.

The only BIG Folder I have that I've not yet placed inside the Master File Server folder is my Master Photo File which is huge, over 250 gigs. I figured I would need to buy another 2 terrabyte drive before I can merge the 500 gig drive with the 500 gig drive that contains pictures and images, and genealogy pictures.

I talked to someone else about this and they could only say RSync when working with large files will spurn off a daemon fork in a new child process, which was no help to me at all.

It didn't seem to do this until my file size crossed the 200 gb point.
Another person said it was dangerous to have only one main folder with everything else inside of that folder.
But isn't that what a directory tree is, one main folder called the tree, and everything else in it?

I've done a few tests where I only changed one document and ran RSync and it took only 4 minutes.
Did another test with 2 changed documents and added 1 new document. Still only took 4 minutes.
Then I changed 3 documents, added 2 new small documents, and 1 small jpg 15k.
Now it bogged down and took 35 minutes to complete.
Doesn't seem it should do that, especially since all the files I changed or added were up in the Family folder, and it doesn't bog down until well into the Reference folder where nothing was changed.

Any ideas?
User avatar
yogi
Posts: 9978
Joined: 14 Feb 2015, 21:49

Re: The Rest Of The Story

Post by yogi »

Thank you for giving my mind something to think about this afternoon. :mrgreen:

First I'll talk about what I am familiar with, which, unfortunately, is not rsync. But, as you know from my past dissertations a little lack of knowledge never stopped me from handing out advice. :mrgreen: :mrgreen:

The old adage about not putting all your eggs into one basket holds true in the world of data backups. For example, instead of a single 2 TB drive, it would be better to have 4-500 MB drives. The obvious reason is that should one drive crash, you don't lose all your data. Another reason to break up large data sets is that they are indexed in tables so that you can see them listed as directories and such. Those tables get to be huge and take a lot of time (relatively speaking) to read recursively each time the process is looking for something specific. What I'm suggesting here is that things can get slower due to size considerations. Also, using 4 drives will allow you to do 4 backups simultaneously. It's true that your main computer has only one processor and one stream of data, but it can set off as many different remote processes as you have connected to your computer. rsync involves two sessions, one on the host and one on the remote, in order to accomplish the transfer. Well, you can do 4 in parallel a bit faster than just one. The downside is cost. You will have to weigh the benefits of your time waiting vs the one time cost of the hardware.

Should 4 drives not be an acceptable solution, then maybe 4 partitions on the same external storage device would work. You would still be vulnerable to the entire device crashing, but with 4 partitions you can do the parallel sync trick just as if you had the hardware.

Think about path names when you create directories and nested sub-directories. The longer the path name is the longer it takes to read it. This might be another reason to break down the massive data into smaller pieces. You would also want to keep the file names as short as possible for the same reason.
I talked to someone else about this and they could only say RSync when working with large files will spurn off a daemon fork in a new child process, which was no help to me at all.
I believe that advice is the technical explanation for what I just told you. LOL

And, speaking of big means slow, in the case of RAM big is better. I would suspect rsync uses buffers all over the place to check out file differences and to do the actual transfers. More RAM means bigger buffers are possible and shorter data transfer times. And, as long as I'm talking about hardware, your external storage is solid state, right? If not you are spending way too much time waiting for things to happen mechanically on the HDD.

And now for the technical explanation:
WikipediA wrote:An rsync process operates by communicating with another rsync process, a sender and a receiver. At startup, an rsync client connects to a peer process. If the transfer is local (that is, between file systems mounted on the same host) the peer can be created with fork, after setting up suitable pipes for the connection. If a remote host is involved, rsync starts a process to handle the connection, typically Secure Shell. Upon connection, a command is issued to start an rsync process on the remote host, which uses the connection thus established. As an alternative, if the remote host runs an rsync daemon, rsync clients can connect by opening a socket on TCP port 873, possibly using a proxy.[24]

Rsync has numerous command line options and configuration files to specify alternative shells, options, commands, possibly with full path, and port numbers. Besides using remote shells, tunnelling can be used to have remote ports appear as local on the server where an rsync daemon runs. Those possibilities allow adjusting security levels to the state of the art, while a naive rsync daemon can be enough for a local network.
https://en.wikipedia.org/wiki/Rsync#:~: ... connection.

mmm ... I'm only being slightly facetious here. The gist of what you are telling me is that the rsync process seems to slow down to a snail's pace when the data transfer size reaches a certain limit. Things start to happen when you get to that point, like daemon processes and forks and the Lords of Linux only know what. There are also security issues built into the rsync command which could build up and slow down the process when size is a factor. Apparently there are options for the rsync command which control those security checks. My guess is you have no idea how to set that up and are allowing the command to do what it will. If you can find the options to turn off all the secure transfer protocols, because you don't need them on your home network, that too would cut down on the transfer time.

And, if all else fails, you can run rsync unattended overnight if you do it with a cron. This would be a very cost effective way to back things up, but I'm guessing you want to be there to supervise things just in case. Well, things can blow up if you are present or not. :grin:

To summarize I would:
  • Break up the large data sets into smaller pieces.
    Rename things and restructure complex directories to shorten the path names.
    Upgrade all storage to solid state. Increase RAM to as much as you can afford.
    Optimize the rsync command line instruction to eliminate unnecessary checks and verification. Also, consider running several rsync commands in parallel.
    Consider unattended backups via cron jobs.

And, as a post script, you must know by now that I am not all that familiar with rsync. I have a feeling there are better and faster ways to accomplish the mirroring with third party software. I know that represents an expense, but it might be worth going with something fully mature and tested in order to save some time and assure no data loss.
User avatar
Kellemora
Guardian Angel
Guardian Angel
Posts: 7494
Joined: 16 Feb 2015, 17:54

Re: The Rest Of The Story

Post by Kellemora »

Wow, that was a lot of great detail Yogi, Thank You!

You should know by now, I would never trust my data to a single HD or any storage device for that matter.
After all I'm the KING of Redundant Backups, hi hi.

Let me jump back in history for a moment. Without mentioning my former off-site storage.
My old File Server was nothing more than a pair of 500 gig external hard drives. One for daily use, and one as a backup of the daily use one. File folders I was working on daily were first downloaded to the internal HD on my computer. As I finished with each one, I would upload it back to the Main File Server External Hard Drive. Then at 3 am every night, the external HD would backup to the 2nd drive. This way I always had two copies of my File Server.

That being said: I did not have a File Folder named Master File Folder.
Instead I had only a few top folders, such as Business, Family, Property, Reference, Websites, and the Master Photo Folder.
And naturally, inside each of those folders were sub folders sometimes nested quite deep.

With my HD set up this way, I had to have a few different lines in my RSync instructions to save to each of them individually.
So at 3am RSync would start Syncing the folder named Business from Drive 1 to Drive 2, then it would move to Family from Drive 1 to Drive 2, etc.
After letting it do it this way for a few years, I found I could just tell RSync to copy the ENTIRE Drive 1 to Drive 2 and not need all those extra lines for each folder. This actually sped up the process considerably.

In my new set-up here, I decided to name my Top Folder Master File Server, and it is the ONLY Folder on Drive 1.
I really don't see why this would make a difference in how long it takes RSync to run, other than previously RSync would only look at the Folder named Business until it was done, then Family, etc. But apparently now it has to look at all of the folders inside of the Master File Server folder first. Although when you watch it run, it is not showing that it is working this way. It shows reading Business, deletes a changed file and adds a new one right back. When done with Business it moves to Family, etc. So it appears to still be running sequentially through the folders and not reading them all at once.

This is why the sudden slow down has befuddled me.
Seems like when RSync is done with the folder named Business, and moved on to the Folder named Family, it would forget about the folder named Business.

However, you may have something about LONG file names might be the cause.
My last folder named Reference does have several files with long file names.
However, when there are no changes, it reads them all in like 4 minutes.

If I get a chance, I'm going to try something today by adding an additional 200 gig folder to the main folder and see if it chokes or does the same slow thing. If it only runs at the same slow pace near the end, I'll assume it is not the file size.
From what I understand, RSync is used to back up entire major databases for huge websites. Takes a lot of time true, due to the size. I've read a lot of the data on the RSync websites and nobody but nobody talks about slow-downs.

Well, except for the obvious of copying a file from the same hard drive to the same hard drive, which is what happens if you want to move a file to a new folder that would take too long using copy n paste. This is where RSync shines.

As an aside: I had some purchased Backup programs, and one of them failed miserably causing my wife to lose years of photo's. The backup program only saved links back to the original drive. When I checked the backup, everything looked OK, so I went ahead and reformatted the drive on her computer to install a new Windows OS. Then when I went to put her stuff back, there was nothing on the backup drive other than links to the original files that were now gone.
I've never trusted proprietary backup software since that time.
User avatar
yogi
Posts: 9978
Joined: 14 Feb 2015, 21:49

Re: The Rest Of The Story

Post by yogi »

I've read good things about the move (mv) command compared to rsync. Since I have no need for using either one, I don't have a lot of experience to pass on to you. It would seem that rsync has more options and can do more than a simple mv. That is both a plus and a detractor. If all you want to do is copy files from one drive to another, then mv would work and I'm guessing it would work faster than rsync because it's simpler. The downside here is that you might have to script several mv commands to get all the backups from the various sources. One of the examples they give shows how to move files selectively, i.e., based on files with the newest timedate stamp. This is what rsync is doing when you mirror one drive to another. I sense that you can eliminate a lot of overhead by using mv instead of rsync. There might be a learning curve to overcome, but maybe not. You do have experience.

And I must agree that not all backup programs are created equal. Backing up by merely using links seems to defeat the purpose of having a backup in the first place. It may be simple and fast, but it's the wrong strategy for anything serious. If you were to purchase a backup program you would want to read the reviews and get as much information about it in advance of sending them any cash. But, as I noted above, I think you are a hands-on kind of guy and probably are better off watching rsync do it's thing live. Oh, and by the way, I am sure there are log files that document what rsync moved, when, and how. Find those files and you may get a better idea of what it's doing. That might inspire you to do it a better way.

And, of course, never trust a backup strategy until you have fully tested it and verified it does what you need to be done. This is particularly true when you do backups for disaster recovery.
User avatar
Kellemora
Guardian Angel
Guardian Angel
Posts: 7494
Joined: 16 Feb 2015, 17:54

Re: The Rest Of The Story

Post by Kellemora »

I've used the move command a few times, and it should be used with the compare and rewrite if the files don't match.
I suggest you never try using it with image files though. It corrupted over half of them.

I steered clear of RSync for eons, because I didn't understand what it actually did. I did not want to keep what I had on one drive Synced up with another drive, at least the way I was thinking it worked.
Turns out it didn't work that way at all. It was basically a copy program that checked your current files to see if any of them changed, and if so, it would only copy the changed files over and delete the original. Didn't like that idea at first either, hi hi. Mainly because if I made a change to a document, I would give it a new sequential name so I had the old one to fall back on. Turned out I could still do that with RSync since all it did was copy new and changed files to the backup drive.

You see, when I think of Sync, I think, If I change something here, it is changed everywhere that is Synced. So it wouldn't matter which drive I changed a file on, they would all be the same. Except that was not how it worked.

If you recall, I used to have my daily working file, which was saved to two external hard drives, then the second external hard drive would be saved to a drive down at my house, so if my office burned down, I still had a copy. At one time I also saved my important stuff to my brothers computer in St. Louis.
For a long time, I was just carrying my backup drive at the house, up to my office, connecting it only long enough to copy my backup drive up here to it, then carrying it back down to the house, manually.
Then I got the NAS that eventually went bad from the lightning strike along with most of my equipment in the office and in the house. Thankfully I had all those old IDE drives laying around with most of the data on them I never erased.
So I was able to recover all but the most current stuff I lost. And even then, I managed to get some stuff off an internal drive so lucked out in a lot of ways.

OH, before I forget. I did add the 200 gig file for a test. Naturally it had to copy that 200 gig file to the backup first which took a while. Then I ran RSync with no changes and it took 5 minutes instead of 4 so no biggie there.
Then I purposely added about a dozen files in different places in numerous folders, and ran RSync again to copy them over to the backup. This time it did not slow down at the end like it did before, even though I purposely added some image files to the 200 gig folders just to see. It only took 7 minutes which is fine considering the file size.

But then, and this is what don't make sense.
I worked on three document files, and two text files. That's it, and they were all in the very first folder named Business.
When I ran RSync before I went home for the evening. It did it's slow down at the end again. Took 22 minutes to copy those 5 files, sorta. It copied them right away when I first started RSync and was just going through all the unchanged stuff. And this is what is bugging me as to why. The files it was dragging it's feet on have no files with long names, and most of them are not very large either. It made it through the image files fairly fast. It bogged down while doing the Reference folder which had no changes in it.
User avatar
yogi
Posts: 9978
Joined: 14 Feb 2015, 21:49

Re: The Rest Of The Story

Post by yogi »

Again I must qualify my response by saying I don't have much experience with rsync. However, just reading what you describe suggests that rsync has to look at all the files you specify to verify which changed and which did not. Those that changed get copied to some kind of buffer or holding area and copied to the synced drive when it's convenient. The process of looking up timedate stamps to determine which files have been changed should go fairly quickly, and it does most of the time. Those times when it drags its feet surely indicates that something more than a simple looking at timedate stamps is going on. That's were I hit the brick wall because I don't know how rsync is actually doing its thing.

System log files spell out what is happening at any given moment. Everything is recorded in Linux. That's why I suggested you find the ones related to the rsync (in /var/log ?) and see what the difference is between the two scenarios, long and short. Also, while I was trying to figure out why it takes so long for Linux to boot I ran across an article about how to troubleshoot such a multi-faceted problem. There were a couple very interesting command line routines that spelled out how long it took (in milliseconds) for a particular process to complete. Once you got a listing of where the slowdown was, then you could look into that specific section of the boot log to see exactly what processes are lagging. It's up to you to figure out why they are lagging and fix it. This was all new to me at the time and I was surprised to learn how well documented the process executions are and how you can use bash to isolate the problem. Something of this nature would be needed to look at the rsync process and determine exactly what is different between the fast copy and the slow copy.
User avatar
Kellemora
Guardian Angel
Guardian Angel
Posts: 7494
Joined: 16 Feb 2015, 17:54

Re: The Rest Of The Story

Post by Kellemora »

OK, dig this:
Assuming I changed absolutely nothing else on my computer.
And my file size is 354 gigs on both the main drive and the backup.
I ran RSync and it took exactly 7 minutes 12 seconds. Of course nothing was changed.
I had 5 files I knew to be identical files but under different names.
I removed four of them from the main drive.
Ran RSync and it took exactly 7 minutes 21 seconds, the extra time was because it deleted those four files.
Ran RSync again with no other changes, this time it took 7 minutes 34 seconds.
I renamed the 1 file I kept previously from Diane-and-Larry to Larry-and-Diane, to put the names in order as they appear.
Ran RSync again, and it took 9 minutes and 55 seconds.
It deleted the file Diane-and-Larry, and then wrote the file Larry-and-Diane right away, so it was done with that part.
It did not slow down until it got to the folder named Property, where nothing was changed, and bogged down there for a tad over a minute, then moved on through the rest super fast.
Ran RSync without doing any other changes, this time it ran back at 7 minutes 18 seconds.

I'm thinking possibly it is not RSync that is slowing down, but possibly my LAN, the times it took over 20 minutes.
I got thinking about that, even though the files are not going over the LAN, they are going from inboard drive to USB drive connected to this computer. But the main slowdown came about the time Debi was down at the house on her computer. Why that would affect anything I don't know. But I've not had that big slowdown since adding the extra 130 gig file. Time will tell what it is I suppose.
User avatar
yogi
Posts: 9978
Joined: 14 Feb 2015, 21:49

Re: The Rest Of The Story

Post by yogi »

I can imagine that buffers and stacks are created while rsync is rsync-ing. Those buffers and stacks have limits as to their size. To accommodate that kind of limitation, because those buffers fill up pretty quick, they use something like a pre-fetch that holds the data before it goes to the buffer. All reads and writes are done by that buffer so that the pre-fetch is what keeps it full and all the data available. When the write process is complete, the pre-fetch empties first and then the buffer and then it all stops. If for whatever reason the buffers fill up, the process stalls. rsync is designed to work over a network with a remote client so that it's not hard to imagine the network (router?) slowing down and affecting rsync in the process. But, something internal to the computers can do it too. My first instinct would be to look at the computer generating the data, but the one receiving it can mess things up too. So, yeah. We are back to the original proposition. Anything can be doing it. LOL

One additional thought about routers slowing down. That usually would happen when somebody on the network is streaming a lot of data and there are other users also trying to do something over the LAN. A router with MIMO technology built in will allow multiple data transfers in parallel; multiple streaming in fact. Routers lacking the MIMO capability can slow down the entire network when one node is heavily active. Most routers have two bands of RF and at least 4 Ethernet ports these days. Putting your rsync computers on one of those bands separate and apart from what Deb is connected to might give you some insight as to where the slowdown is occurring.
User avatar
Kellemora
Guardian Angel
Guardian Angel
Posts: 7494
Joined: 16 Feb 2015, 17:54

Re: The Rest Of The Story

Post by Kellemora »

I did notice a slowdown on our LAN after DirecTV changed their box to a WiFi type.
I think some of the shows the frau watches is coming over the Internet and not through the satellite dish.
I know when she is recording some shows they are.
Plus now I have the WiFi Access Point hardwired to the LAN, which is also a WiFi router.

I have a heck of a time setting up RSync to use over the LAN, have to make sure the end drive is Mounted in the right way.
And once I get it all set up and working, I forget how to do it by the next time I want to set one up.
And the really sad thing is, if I don't mount the remote drive before I run RSync it shows it worked when it didn't.
It only saves to the space on the local hard drive that it thinks is Synced to the remote.
This happens when the IP address changes due to a power outage and reboot.
So many things to double and triple check before you attempt doing anything over the LAN.
They should make it work using simple file sharing. If you have the folder mounted on your desktop, it should just work, period. But it don't work that way!

Hey, if you thought I was crazy with my redundant backups.
I think I've met my match. Another person with a Phobia to NAS Controllers that can go bad.
Was talking to a fellow who works at one of our local radio stations last night.
He has four LARGE NAS units, each one a different brand, and each with 4 terrabytes I think he said.
All Four are used to store the exact same data.
Whenever he adds or changes a file on any of his computers, he backs it up to each of those NAS units.
He used to only use two identical brand NAS units, but when one of them went bad on him, he worried about the other one going out too. This is why he now has four units each a different brand.
He also does not trust his data to any external cloud servers!
User avatar
yogi
Posts: 9978
Joined: 14 Feb 2015, 21:49

Re: The Rest Of The Story

Post by yogi »

There are a whole lot of ways to configure backup schemes to suit one's own personal paranoia. Redundancy is a requirement when you get serious. Four NAS storage devices to do the same job seems like trouble waiting to happen. The best redundancy would be to backup to different media at each layer. These days storage media is pretty cheap and it would not be unreasonable to mirror data servers similar to what you are currently doing. Back that up to NAS, and also back that up to removable solid state drives, and also back that up to the cloud. And for the extremely paranoid among us, yet another backup off site, preferably more than 300 miles distant. If none of that allows you to sleep at night, then check out the possibility of acquiring mass quantities of Valium.

I don't know how Dish works but if they connect to your WiFi you need two separate LAN's in your house. One LAN would be dedicated to multimedia while the other would be dedicated to computer networking. One of the houses we looked at before buying this one had a theater in the basement. It was run by a server as big as any I've seen at Motorola. LOL I personally think you would benefit from separate LAN's as opposed to running routers in tandem. I only have 7 connections to my router at the moment. It's a slow day in O'Fallon. :lol:
User avatar
Kellemora
Guardian Angel
Guardian Angel
Posts: 7494
Joined: 16 Feb 2015, 17:54

Re: The Rest Of The Story

Post by Kellemora »

You would die if you saw how I have things connected here, hi hi.
ONE port in my Router is handling EVERYTHING in my Office.
I have an 8 port Switch up here, and one of those ports is used by a 4 port Switch.
Another by the WiFi Access Point, another by a Printer, that is also connected via USB.
A single CAT5E cable, running parallel with a TV Coax, Electric Wire, and two Telephone Lines are bundled in a wrapped sheath to protect from the elements.
If that don't cause a slowdown from cross-interference I don't know what will.
Yet according to my meters, I'm getting my gigabit speed through it.

I thought after the hailstorm we replaced the CAT5E with CAT6, but apparently it was only replaced with CAT5E again, only at CAT6 prices hi hi.
User avatar
yogi
Posts: 9978
Joined: 14 Feb 2015, 21:49

Re: The Rest Of The Story

Post by yogi »

I don't think the copper core of the Ethernet cable is any different in any of the categories. It's all in the shielding and possibly the diameter of the conductors. From what I recall of the specs the only difference between CAT 5 and CAT 6 is the transmission speed. There is a CAT 7 now too, I believe, which is what you would need for gigabyte streaming. Maybe you don't need two LAN's after all. You probably need three or more. LOL
User avatar
Kellemora
Guardian Angel
Guardian Angel
Posts: 7494
Joined: 16 Feb 2015, 17:54

Re: The Rest Of The Story

Post by Kellemora »

Going back a number of years, a few of us had TWIN 56k modems, and of course phone lines.
But only a couple of ISPs offered the two-tier systems.
Since I didn't do gaming, although I had the twin modems I rarely used them at the same time.
Only when I was uploading pictures to the website and that was about it.
Post Reply