The database is now being used to store your visited topics.

Banshee

Hello everyone. We are slowly working on the background to make the PPM cookies a bit less heavy on your disk. As some of you may know, browsers limit cookie size up to 4kb. Many members, including yours truly, were storing so much data on cookies (most of it being the visited or unread topics/forums) that were having problems with this limit. After you reach the limit, you don't know exactly what is written and what is not. So, some things may not work as expected. I'm not gonna get into details about what kind of problems you could have, but the easiest one to track is when you read the topic and it still shows as unread. Unfortunately, it can also cause much worse problems than that. So, we have decided to store this data on the database.

I don't know all the effects of this change yet and how it affects your browsing experience, especially if you start browsing the forum as a guest and eventually log in. What I know is that some topics that you've read might show as unread and some topics you may have not read could be shown as read. It might be annoying at first, but things might return to normality eventually.

Sorry for any inconvenience that this may cause. This change is not a complete fix for this situation, since some of you may still have cookies that bypass the 4kb limit. For those concerned about changes on PPM's security as a consequence of this measure, we recommend two things.

1) Delete all your PPM cookies (all domains) and log in again.
2) Do not give away your SID when you post any sort of link to PPM. We are trying to minimize the occurrence of these events, but your help will be truly appreciated.

Key Words: #PPMForums #ProjectPerfectMod #Cookies #Security

MadHQ · Commander **Joined**: 07 Nov 2003

I came to the forms and it was all "unread" I cant even see "new" post's. This is an interesting change...

_________________
MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.)

G-E · Defense Minister **Joined**: 09 Feb 2015

Perhaps you could make an update/transition page a user could visit to have their cookie read and populate the database one last time before redoing the cookie to minimalism?

_________________
http://www.moddb.com/mods/scorched-earth-ra2-mod-with-smart-ai

Banshee · Posted: Sun Jun 06, 2021 5:46 am Post subject:

That's a good idea, G-E, if I had time to actually work on it. However, I have some tough real-life deadlines related to my Ph. D., which is not allowing me to spend much time coding PPM's forum software recently. So, I have to deal with this problem by prioritizing what is on my range. And, right now, this is what I can do so far.

MadHQ · Commander **Joined**: 07 Nov 2003

As long as some one clicks "Mark all forums read" from the main forum page its not so bad.

_________________
MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.)

Banshee · Posted: Tue Jun 08, 2021 9:09 pm Post subject:

Update: I'm still slowly tweaking how the forum deals with cookies on all affected domains. But the idea is to actually minimalize cookies instead of doing what G-E suggested, for the sake of simplicity, predictability, and maintenance.

MadHQ · Commander **Joined**: 07 Nov 2003

The "Mark all forums read" is not working. Sad

_________________
MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.)

Banshee · Posted: Wed Jun 23, 2021 8:18 pm Post subject:

It worked here. I can't reproduce any problem with "Mark all forums read" here.

MadHQ · Commander **Joined**: 07 Nov 2003

It works when its clicked, but when browser is closed and reopened it goes back to being unread. If you are using a browser that is always running behind the scene you many have to incompletely close it.

_________________
MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.)

Banshee · Posted: Thu Jun 24, 2021 3:44 am Post subject:

Sorry, but it doesn't make any sense. The topic visited data is now stored in the database. Not on the browser. What could possibly be the cause of your problem would be the cache from the browser, don't you think?

E1 Elite · General **Joined**: 28 May 2013

It works for current session. Once you logout and clear history/cache and return, it resets it back to unread. May be it doesn't update the DB.

Banshee · Posted: Thu Jun 24, 2021 6:23 pm Post subject:

I will take a look at it as soon as possible.

MadHQ · Commander **Joined**: 07 Nov 2003

There is some thing wrong with PHP. I have a very simple page counter, that reads/writes a file. Way to frequently it resets the value. Its almost like memory is flushed.

I do lock the file. Banshee can you look at my PHP script and see if I am doing anything wrong. script name = grave_counterControl.php, its located in the main directory. In the past I had to create a backup counter file that I would use to check the value and make sure its correct, if not, correct it. Now it almost seems like I have to go a 3rd check to verify...

A counter should not be this complicated.

_________________
MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.)

Banshee · Posted: Mon Jun 28, 2021 8:27 am Post subject:

Your counter script works correctly, as far as I could test it with a dummy test file (you might see a file with an unusual name at the same directory). Now I'm checking the rest of the files to look for any kind of interference that could be causing this problem.

Update: It seems to be working fine here. I have tested it with the latest Mozilla Firefox. I'll now dump all test files.

MadHQ · Commander **Joined**: 07 Nov 2003

How about this... In the past, trying to identify the issue I have a put in a email notification system so when it fails, it sends an email.

I can set it up so it sends an email to me and you. I will warn you... In the past I have seen that result in hundreds of emails being sent.

What do you think, team work? I have limited access to PHP run time so I cant do much...

If you want to try, and have a specific email address you would like me too put in the script, I can add it and we would have to wait for it to happen again.

_________________
MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.)

Banshee · Posted: Tue Jun 29, 2021 5:26 am Post subject:

You don't need to do that.

It will have this flush effect if, for reasons that usually go beyond our comprehension, it fails to read the file. It either happens when it creates the counter file in the first run (but that wouldn't be a problem, since the counter variable would be 0 anyway) or if the server resources are overloaded. In this new server, it is a much more rare event than in the previous one.

MadHQ · Commander **Joined**: 07 Nov 2003

It did it again. It seems like its more frequent on this new server.

Screenshot from 2021-06-29 18-19-49.jpg

Description:

Filesize:

67.3 KB

Viewed:

4801 Time(s)

_________________
MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.)

Banshee · Posted: Wed Jun 30, 2021 12:22 am Post subject:

I don't know how reliable flock can be when you use it with a webserver. Also, its behavior varies depending on the system.

If you test it using your system alone, your code works. You won't find any problems. If a bunch of bots decides to crawl your site or do a denial of service on it, things can get messed up. And hordes of bots haunting sites is more common than you think. PPM suffers from this evil quite often, regardless of our server.

Apache servers do have a "time to leave" policy (scripts don't run forever) and PHP may unlock your semaphore when the script finishes running (in case you don't unlock it before).

So, imagine that a bunch of bots decide to crawl your site nearly at the same time and your scripts start to get stuck on the semaphores. What happens? It would be possible that a script could be interrupted while reading or writing the counter file, leaving the semaphore closed to the other instances until PHP decided that the script has finished running and opened the semaphore on its own. If the script is interrupted while writing the file, it could generate an empty file and other instances of the script would reset the counter.

If you add a "backup semaphore", as you did, you increase the odds of this kind of event happening. If you make it send an email, the odds are even higher. So, sometimes, it is better to keep things simple or simply not count when a bunch of bots decided to crawl your file at the same time.

MadHQ · Commander **Joined**: 07 Nov 2003

There is some thing odd going on... My counter has jumped a few hundred...

This is odd...

Sounds like you are saying remove the back file... I could try that...

Screenshot from 2021-06-29 21-29-26.jpg

Description:

Filesize:

67.25 KB

Viewed:

4789 Time(s)

_________________
MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.)

Banshee · Posted: Wed Jun 30, 2021 4:49 am Post subject:

And I'm also telling you to protect your downloads (and counter) against bots.

MadHQ · Commander **Joined**: 07 Nov 2003

Wow... Maybe PPM (or myself) we need to set up some kind of anti-bot thing...

In about 24 hours... My counter has gone up about 4000...

My site does not get that normal much traffic.

Screenshot from 2021-06-30 17-45-06.jpg

Description:

Filesize:

25.95 KB

Viewed:

4711 Time(s)

_________________
MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.)

MadHQ · Commander **Joined**: 07 Nov 2003

I removed the backup thing I did, and I did some google searching how to identify bots via PHP, and I came up with this:

Code:

$fileNameMain="grave_counter.txt";
if(stripos($_SERVER['HTTP_USER_AGENT'],'bot') === false &&
   stripos($_SERVER['HTTP_USER_AGENT'],'crawl') === false &&
   stripos($_SERVER['HTTP_USER_AGENT'],'slurp') === false &&
   stripos($_SERVER['HTTP_USER_AGENT'],'spider') === false){
$valueFooter=intval(readCounterFile($fileNameMain))+1;
writeCounterFile($fileNameMain,$valueFooter);
} else {
writeCounterFile($fileBotNameMain,$valueBotFooter);
}

This should stop some of the bots from reading/writing to my counter. I also set up a counter just to count the bots... LOL seems like only after a few hours I can tell some thing "bot-ish" is hitting my site.

I guess I will have to watch and see if it helps...

Screenshot from 2021-07-04 19-12-22.jpg

Description:

Filesize:

17.25 KB

Viewed:

4663 Time(s)

_________________
MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.)

G-E · Defense Minister **Joined**: 09 Feb 2015

Just fyi I was greeted by a field too large error when visiting the site today, I tried a few times figuring it was an outage, but clearing cookies fixed it.

_________________
http://www.moddb.com/mods/scorched-earth-ra2-mod-with-smart-ai

MadHQ · Commander **Joined**: 07 Nov 2003

Unfortunately my attempt failed. I guess I will have try deeper, or find a different plan. The page counter restarted...

There is some thing wrong with the config(s) on this server... This is not new, this has been an ongoing issue.

Also the unread post still is not working correct...

_________________
MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.)

MadHQ · Commander **Joined**: 07 Nov 2003

MJ12 bot is being a pain in the back side!!! I have tried many ways to stop it, but it will not stop... It keeps hitting my site. It doesn't seem to acknowledge robot.txt... Or do I have robot.txt set up wrong? Nor is my strpos finding it... Its really bizarre... its just a string and for some reason '.com' or 'bot' cannot be found... Maybe there is some kind of binary data interfering...

I must kill this MJ12 bot!!

Screenshot from 2021-07-11 17-48-47.jpg

Description:

Filesize:

191.58 KB

Viewed:

4542 Time(s)

_________________
MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.)

Banshee · Posted: Mon Jul 12, 2021 12:12 am Post subject:

First of all, MJ12 official site for reference's sake:
https://www.mj12bot.com/

Apparently, your robots.txt was supposed to block MJ12, according to their official site, if their bot is coded correctly. However, to be honest, I am not really confident about that bolded part. So, for MJ12 and few other questionably coded bots, I do recommend you add a flood control module to your site to prevent their access if either the IP or the user agent are in your bad bot list or simply record its last access time and prevent their access if a number of seconds determined by you have not passed yet. I had to do that to PPM front page and forums and MJ12 is filtered there where I filter all user agents with "bot" in their name.

Update: I've added a file in your FTP root where you can use as a base for a flood control module, if you need. You'll find it by listing the files by modified date Wink

.

MadHQ · Commander **Joined**: 07 Nov 2003

Back on topic... LOL

Banshee, did you do some thing to fix the read post's? It seems better.

I still have to tinker with my site... Not had a chance to deal with the bots just yet... its a WIP...

_________________
MadHQ's Graveyard - Click here!
(Permissions) - (F.A.Q.)

Banshee · Posted: Sun Jul 18, 2021 9:31 am Post subject:

Nope, I haven't changed anything recently.

E1 Elite · General **Joined**: 28 May 2013

It got fixed for most part now. There is some minor stuff like after posting and logout, that post is shown as unread in guest mode immediately after logout. It gets cleared later, so not an issue.